---

base_model: coder3101/gemma-4-31B-it-heretic

license: gemma

library_name: gguf

pipeline_tag: image-text-to-text

tags:

- gemma4

- gguf

- llama.cpp

- quantized

- q8_k_xl

- imatrix

- vision

- heretic

- abliterated

---

gemma-4-31B-it-heretic-Q8_K_XL-GGUF

Imatrix-calibrated Q8_K_XL GGUF of coder3101/gemma-4-31B-it-heretic, built for llama.cpp. Includes the full BF16 vision projector so the model retains image understanding.

Heretic is an abliterated variant of google/gemma-4-31B-it that suppresses refusal behavior while preserving instruction-following. The abliteration survives quantization cleanly — post-quant tests show the model still answers directly without hedging.

Files

| File | Size | Notes |

| --- | --- | --- |

| gemma-4-31B-it-heretic-Q8_K_XL.gguf | ~33 GB | Language model weights (Q8_0 base + selective F16) |

| mmproj-BF16.gguf | ~1.2 GB | Vision projector, full BF16 |

Recipe

Base format is Q8_0 with the token embedding and output tensors also at Q8_0. Six transformer blocks identified as activation-outlier hotspots are escalated to F16 on their attention Q/K and full MLP (gate / up / down):

blk.{1,52,53,57,58,59}.attn_q.weight    = f16
blk.{1,52,53,57,58,59}.attn_k.weight    = f16
blk.{1,52,53,57,58,59}.ffn_gate.weight  = f16
blk.{1,52,53,57,58,59}.ffn_up.weight    = f16
blk.{1,52,53,57,58,59}.ffn_down.weight  = f16

This mirrors the bit-allocation pattern used by Unsloth's UD-Q8_K_XL variants: protect the first transformer block and the last few blocks where activation magnitudes blow up, leave the rest at Q8_0 where the K-quant superblock structure handles things just fine.

Calibration

Imatrix was computed from a 613K-token mixed corpus:

• bartowski/c4_datasets_calibration_data calibration_datav3.txt — general text, ~70K tokens
• eaddario/imatrix-calibration tools_micro.parquet — tool-calling patterns, ~400K tokens
• eaddario/imatrix-calibration code_micro.parquet — code instructions, ~150K tokens

Imatrix run: 282 chunks at --ctx 2048 --batch 512, full GPU offload, on 4× A100 80GB.

Why K-quants instead of GPTQ

Gemma 4's MLP blocks have heavy-tailed activation distributions. GPTQ's per-column Hessian update fails Cholesky inversion on every down_proj even with aggressive dampening, falls back to RTN, and produces visible token-level corruption in code output (background-color: #3-, dropped articles, etc.) at every bit width tested (W4A16 sym/asym, W8A16, mixed). llama.cpp's K-quant superblock-scale format plus per-channel imatrix weighting handles the outliers correctly. Validated with code generation, reasoning, and tool-calling tests post-quant.

Running with llama.cpp

llama-server \
  -m gemma-4-31B-it-heretic-Q8_K_XL.gguf \
  --mmproj mmproj-BF16.gguf \
  -ngl 999 \
  -c 262144 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --chat-template-kwargs '{"enable_thinking":true}' \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --host 0.0.0.0 --port 8081

Tested on a 4× 24 GB rig (1× RTX 4090 + 3× RTX 3090) with 256K context, q8_0 KV cache, flash attention on. The Q8_0 KV cache is the only thing that makes the long context fit.

Caveats

• This is an abliterated model. It will follow instructions it would normally refuse. Use accordingly.
• Vision projector is full BF16; if you need a smaller projector, use an external Q8_0 mmproj at the cost of some image-understanding quality.
• The Q8_K_XL label follows Unsloth's naming convention for "Q8 base with selected layers escalated to higher precision" — the actual ggml type of the base tensors is q8_0.

Credits

• google/gemma-4-31B-it — base model
• coder3101/gemma-4-31B-it-heretic — abliteration
• unsloth — recipe pattern this quant clones
• bartowski and eaddario — calibration corpora
• llama.cpp — quantization toolchain