FreedomAISVR/Gemma-4-12B-it-QAT-NVFP4-GGUF overview
Gemma 4 12B Instruct QAT — NVFP4 GGUF NVFP4 4 bit GGUF of Google's official QAT release, rebuilt from google/gemma 4 12B it qat q4 0 unquantized and verified o…
Runs locally from ~167.0 MB disk (4 GB VRAM class GPUs with llama.cpp / guIDE).
Repository Files & Downloads
Model Details
| Model ID | FreedomAISVR/Gemma-4-12B-it-QAT-NVFP4-GGUF |
|---|---|
| Author | FreedomAISVR |
| Pipeline | — |
| License | apache-2.0 |
| Base model | google/gemma-4-12B-it-qat-q4_0-unquantized |
| Last modified | 2026-10-16T12:33:56.000Z |
Model README
---
language:
- en
- multilingual
tags:
- gemma
- gemma-4
- qat
- nvfp4
- blackwell
- gguf
- vision
- multimodal
license: apache-2.0
base_model: google/gemma-4-12B-it-qat-q4_0-unquantized
---
Gemma 4 12B Instruct QAT — NVFP4 GGUF
NVFP4 (4-bit) GGUF of Google's official QAT release, rebuilt from google/gemma-4-12B-it-qat-q4_0-unquantized and verified on llama.cpp with vision and 256K context at KV Q8_0.
What's in this build
- Quantization: NVFP4 (
general.file_type = 39) — NVIDIA Blackwell 4-bit E2M1 values with E4M3 scaling per 16 values plus an FP32 tensor scale. - Tensor layout: 667 tensors — 328 NVFP4, 338 F32 (norms/biases), and the token embedding matrix kept at Q8_0 (a 262,144 × 3,840 matrix, where 4-bit is a bad trade).
- Params: 11,907,350,576
- BPW: 4.85 (bits per parameter, whole file)
- No hybrid trick: every transformer weight is genuinely NVFP4. Norms stay F32 because quantizing them buys almost nothing and costs accuracy.
Files
| File | Size | Description |
|------|------|-------------|
| gemma-4-12b-it-qat-nvfp4.gguf | 7,219,672,864 B (7.22 GB) | Text model, NVFP4 + Q8_0 embeddings, F32 norms |
| mmproj-gemma-4-12b-it-qat.gguf | 175,115,616 B (175 MB) | Vision projector, F32/BF16 — byte-identical to Google's official mmproj |
The mmproj is shipped at F32/BF16 on purpose. Converting it to F16 breaks vision (see Limitations).
Verified results
Tested on llama.cpp b1868 (4fea119d), RTX 5060 Ti 16 GB, CUDA 13.2:
Vision — OCR test image with five lines including PART-NUMBER: XK-9942, LOT: 7734-B, and 12 + 37 = 49. Output:
GEMMA VISION TEST
PART-NUMBER: XK-9942
LOT: 7734-B
Q4_0 or NVFP4?
12 + 37 = 49
49
All five lines transcribed exactly, arithmetic result correct.
256K context with KV cache at Q8_0 — -c 262144 -ctk q8_0 -ctv q8_0:
- KV cache:
K (q8_0) 1088 MiB + V (q8_0) 1088 MiBover 262,144 cells (8 layers), plus 850 MiB SWA cache (5,120 cells, 40 layers) - Projected 12,953 MiB of 15,174 MiB free device memory → fits with ~2.2 GB headroom
- 257,620-token prompt processed and answered, exit code 0
- Prompt 788.3 t/s, generation 19.5 t/s
- Quantized V cache automatically enables flash attention (required by llama.cpp)
Usage
# text
llama-cli -m gemma-4-12b-it-qat-nvfp4.gguf -ngl 99 --jinja
# 256K context, KV cache at Q8_0
llama-server -m gemma-4-12b-it-qat-nvfp4.gguf -ngl 99 --jinja \
-c 262144 -ctk q8_0 -ctv q8_0
# vision / multimodal
llama-mtmd-cli -m gemma-4-12b-it-qat-nvfp4.gguf \
--mmproj mmproj-gemma-4-12b-it-qat.gguf \
--image photo.png -ngl 99 --jinja
--jinja is required
Gemma 4's chat template cannot be expressed in llama.cpp's legacy template converter. Without --jinja, llama-mtmd-cli throws an uncaught C++ exception during chat-template setup and dies with silent exit code 0xC0000409 (-1073740791) — no error message, no output. With --jinja the template renders correctly (<|turn>…<turn|>, <|think|>).
Limitations
Do not regenerate the mmproj with a plain F16 conversion. F16 conversion of v.patch_embd.weight (6912×3840) overflows F16 accumulation during the patch-embedding matmul. Observed outcomes on this exact model:
- F16 mmproj → repeated
<unused49>tokens, or - F16 mmproj → silent hallucination (model invents image content instead of reading it)
The F32/BF16 mmproj shipped here is unaffected. This matches upstream llama.cpp issue #24146 (closed not-planned, unfixed), root cause 2: "The patch_embeddings_0 matmul overflows F16 accumulation… forcing F32 accumulation fixes it."
Upstream #24146 root cause 1 (resize not filling the soft-token budget) is also unfixed and was not worked around here; it affects token-count accuracy for images, not correctness of this build's OCR results.
Architecture: Gemma4UnifiedForConditionalGeneration, 48 layers, hidden 3840, FFN 15360, 16 KV heads, sliding-window (1024) + full attention in a 40:8 per-layer pattern (matching the KV cache: 40 SWA layers at 5,120 cells + 8 full layers at 262,144 cells). Native 262,144-token context (gemma4.context_length = 262144). Thinking is enabled by default via the <|think|> token.
Source
- Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
- Converter: llama.cpp
convert_hf_to_gguf.py, build b1868 (4fea119d) - Quantizer:
llama-quantizeNVFP4 (locally patched build; NVFP4 is not in stock llama.cpp) - Chat template: native Gemma 4, taken from the source
chat_template.jinja
License
Apache 2.0 (same as the base model).
Run FreedomAISVR/Gemma-4-12B-it-QAT-NVFP4-GGUF with guIDE
Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.
Source: Hugging Face · Compare models