---

language:

• en
• multilingual

tags:

• gemma
• gemma-4
• qat
• mxfp4
• blackwell
• gguf
• vision
• multimodal

license: apache-2.0

base_model: google/gemma-4-12B-it-qat-q4_0-unquantized

---

Gemma 4 12B Instruct - QAT + MXFP4 Hybrid GGUF

QAT-optimized weights preserved at Q4_0, overhead tensors quantized to MXFP4.

What Makes This Different

This is a hybrid quantization of Google official QAT (Quantization-Aware Training) model. Instead of requantizing the Q4_0 weights (which breaks QAT benefits and vision quality), we:

1. Kept all 329 weight tensors at Q4_0 - attention, FFN, embeddings - exactly as Google trained them
2. Quantized only the 338 F32 norm/bias tensors to MXFP4 - these are the overhead tensors (layer norms, RMS norms, etc.)
3. Used Google QAT mmproj (175 MB) - the vision projector trained alongside the QAT model

Why Standard MXFP4 from QAT Breaks Vision

Google QAT model was specifically trained to be resilient to Q4_0 quantization patterns. The weight values learned during QAT compensate for Q4_0 rounding. When you requantize Q4_0 -> F32 -> MXFP4:

• A second round of quantization error is introduced that QAT training did not account for
• MXFP4 uses different block sizes, rounding, and grouping than Q4_0
• Vision tokens flow through the same attention/FFN layers - precision loss disproportionately degrades vision
• The result: models that can not recognize text in images

How the Hybrid Approach Works

Using llama-quantize --tensor-type-file with --allow-requantize:

`

llama-quantize --allow-requantize --tensor-type-file keep_q4.txt input.gguf output.gguf MXFP4

`

The tensor-type-file lists all Q4_0/Q4_K_S tensors to keep at their current type. When the quantizer sees cur_type == new_type, it copies the tensor data as-is - zero precision loss. Only the remaining F32 tensors are quantized to MXFP4.

File Sizes

| Component | Original | Hybrid |

|-----------|----------|--------|

| Weight tensors (Q4_0) | 6.56 GB | 6.56 GB (unchanged) |

| Norm tensors (F32) | ~75 MB | ~15 MB (MXFP4) |

| Total GGUF | 6.65 GB | 6.39 GB |

| mmproj (QAT BF16) | 175 MB | 175 MB |

Why BPW Shows 4.50

The reported 4.50 BPW is misleading - it averages the Q4_0 weights (~4.34 BPW) with the MXFP4 norms (~4 BPW). The actual quality is determined by the Q4_0 weights, which are QAT-optimized.

Usage

`ash

llama.cpp

llama-server -m gemma-4-12b-it-qat-mxfp4.gguf --mmproj mmproj-gemma-4-12b-it-qat.gguf -ngl 99

`

Source

• Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
• Quantized with: llama.cpp build 537 (commit d2c6795)
• Chat template: Native Gemma 4 (thinking enabled by default via <|think|> token)
• Vision: Full multimodal support via QAT mmproj

Files

| File | Size | Description |

|------|------|-------------|

| gemma-4-12b-it-qat-mxfp4.gguf | 6.39 GB | Q4_0 weights + MXFP4 norms |

| mmproj-gemma-4-12b-it-qat.gguf | 175 MB | QAT vision projector (BF16) |

License

Apache 2.0 (same as base model)