---

license: apache-2.0

base_model:

- google/gemma-4-12B-it-qat-q4_0-unquantized

- google/gemma-4-12B-it-qat-q4_0-unquantized-assistant

tags:

- gguf

- llama.cpp

- gemma-4

- qat

- mtp

- vision

- multimodal

- speculative-decoding

- q5_k_l

---

<div align="center">

<img src="https://ai.google.dev/gemma/images/gemma4_banner.png" alt="Gemma 4 Banner" width="100%">

<br>

<p align="center">

<a href="https://huggingface.co/collections/google/gemma-4" target="_blank"><b>Hugging Face Collection</b></a> •

<a href="https://github.com/google-gemma" target="_blank"><b>GitHub</b></a> •

<a href="https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/" target="_blank"><b>DeepMind Blog</b></a> •

<a href="https://ai.google.dev/gemma/docs" target="_blank"><b>Documentation</b></a>

</p>

</div>

Gemma-4-12B-it-QAT-Q5_K_L-MTP-GGUF

High-precision Q5_K_L GGUF quantizations of Google's official Quantization-Aware Trained (QAT) Gemma 4 12B model, complete with its official Multi-Token Prediction (MTP) drafter head and Multimodal Projector (mmproj) for vision and audio understanding.

Repository Files

| Filename | Role | Size | Description |

| :--- | :--- | :--- | :--- |

| gemma-4-12B-it-qat-Q5_K_L.gguf | Target Model | ~9.1 GB | Base model with Q5_K_M attention/MLP and lossless Q8_0 embeddings & LM head. |

| mtp-gemma-4-12B-it-qat-Q5_K_L.gguf | MTP Drafter | ~460 MB | Official QAT-matched MTP drafter (arch: gemma4-assistant) in lossless Q8_0. |

| mmproj-gemma-4-12B-it-qat-Q5_K_L.gguf | Vision / Audio | ~175 MB | Unified multimodal projector for image analysis, OCR, and document understanding. |

---

Running with Vision + MTP Acceleration

To run the model with both image understanding and 2x speculative decoding, supply both --mmproj and --spec-draft-model:

1. OpenAI-Compatible Server (llama-server)

llama-server \
  -m gemma-4-12B-it-qat-Q5_K_L.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-Q5_K_L.gguf \
  --spec-draft-model mtp-gemma-4-12B-it-qat-Q5_K_L.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --ubatch-size 2048 \
  -c 32768 \
  -ngl 99

> Note on Image Inputs: Gemma 4 vision uses non-causal attention across visual tokens, which must fit within a single processing chunk. Always include --ubatch-size 2048 when processing high-resolution images to prevent batch-overflow assertion errors.

2. Interactive CLI with Image Input (llama-cli)

llama-cli \
  -m gemma-4-12B-it-qat-Q5_K_L.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-Q5_K_L.gguf \
  --spec-draft-model mtp-gemma-4-12B-it-qat-Q5_K_L.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --image path/to/document.png \
  -p "Extract all text and tabular data from this image."