GraySoft
Projects Models Compare Cloud benchmarks FAQ Download guIDE →
Model Intelligence Sheet

FreedomAISVR/Gemma-4-12B-it-QAT-NVFP4-GGUF overview

Gemma 4 12B Instruct QAT — NVFP4 GGUF NVFP4 4 bit GGUF of Google's official QAT release, rebuilt from google/gemma 4 12B it qat q4 0 unquantized and verified o…

ggufgemmagemma-4qatnvfp4blackwellvisionmultimodalenmultilingualbase_model:google/gemma-4-12B-it-qat-q4_0-unquantizedbase_model:quantized:google/gemma-4-12B-it-qat-q4_0-unquantizedlicense:apache-2.0endpoints_compatibleregion:usconversational

Runs locally from ~167.0 MB disk (4 GB VRAM class GPUs with llama.cpp / guIDE).

Downloads
1,051
Likes
0
Pipeline
—

Repository Files & Downloads

2 GGUF files detected
Direct downloads for local inference
FileTypeQuantizationSizeLink
gemma-4-12b-it-qat-nvfp4.ggufGGUFGGUF6.72 GBDownload
mmproj-gemma-4-12b-it-qat.ggufGGUFGGUF167.0 MBDownload

Model Details

Model IDFreedomAISVR/Gemma-4-12B-it-QAT-NVFP4-GGUF
AuthorFreedomAISVR
Pipeline—
Licenseapache-2.0
Base modelgoogle/gemma-4-12B-it-qat-q4_0-unquantized
Last modified2026-10-16T12:33:56.000Z

Model README

---

language:

  • en
  • multilingual

tags:

  • gemma
  • gemma-4
  • qat
  • nvfp4
  • blackwell
  • gguf
  • vision
  • multimodal

license: apache-2.0

base_model: google/gemma-4-12B-it-qat-q4_0-unquantized

---

Gemma 4 12B Instruct QAT — NVFP4 GGUF

NVFP4 (4-bit) GGUF of Google's official QAT release, rebuilt from google/gemma-4-12B-it-qat-q4_0-unquantized and verified on llama.cpp with vision and 256K context at KV Q8_0.

What's in this build

  • Quantization: NVFP4 (general.file_type = 39) — NVIDIA Blackwell 4-bit E2M1 values with E4M3 scaling per 16 values plus an FP32 tensor scale.
  • Tensor layout: 667 tensors — 328 NVFP4, 338 F32 (norms/biases), and the token embedding matrix kept at Q8_0 (a 262,144 × 3,840 matrix, where 4-bit is a bad trade).
  • Params: 11,907,350,576
  • BPW: 4.85 (bits per parameter, whole file)
  • No hybrid trick: every transformer weight is genuinely NVFP4. Norms stay F32 because quantizing them buys almost nothing and costs accuracy.

Files

| File | Size | Description |

|------|------|-------------|

| gemma-4-12b-it-qat-nvfp4.gguf | 7,219,672,864 B (7.22 GB) | Text model, NVFP4 + Q8_0 embeddings, F32 norms |

| mmproj-gemma-4-12b-it-qat.gguf | 175,115,616 B (175 MB) | Vision projector, F32/BF16 — byte-identical to Google's official mmproj |

The mmproj is shipped at F32/BF16 on purpose. Converting it to F16 breaks vision (see Limitations).

Verified results

Tested on llama.cpp b1868 (4fea119d), RTX 5060 Ti 16 GB, CUDA 13.2:

Vision — OCR test image with five lines including PART-NUMBER: XK-9942, LOT: 7734-B, and 12 + 37 = 49. Output:

GEMMA VISION TEST
PART-NUMBER: XK-9942
LOT: 7734-B
Q4_0 or NVFP4?
12 + 37 = 49

49

All five lines transcribed exactly, arithmetic result correct.

256K context with KV cache at Q8_0 — -c 262144 -ctk q8_0 -ctv q8_0:

  • KV cache: K (q8_0) 1088 MiB + V (q8_0) 1088 MiB over 262,144 cells (8 layers), plus 850 MiB SWA cache (5,120 cells, 40 layers)
  • Projected 12,953 MiB of 15,174 MiB free device memory → fits with ~2.2 GB headroom
  • 257,620-token prompt processed and answered, exit code 0
  • Prompt 788.3 t/s, generation 19.5 t/s
  • Quantized V cache automatically enables flash attention (required by llama.cpp)

Usage

# text
llama-cli -m gemma-4-12b-it-qat-nvfp4.gguf -ngl 99 --jinja

# 256K context, KV cache at Q8_0
llama-server -m gemma-4-12b-it-qat-nvfp4.gguf -ngl 99 --jinja \
  -c 262144 -ctk q8_0 -ctv q8_0

# vision / multimodal
llama-mtmd-cli -m gemma-4-12b-it-qat-nvfp4.gguf \
  --mmproj mmproj-gemma-4-12b-it-qat.gguf \
  --image photo.png -ngl 99 --jinja

--jinja is required

Gemma 4's chat template cannot be expressed in llama.cpp's legacy template converter. Without --jinja, llama-mtmd-cli throws an uncaught C++ exception during chat-template setup and dies with silent exit code 0xC0000409 (-1073740791) — no error message, no output. With --jinja the template renders correctly (<|turn>…<turn|>, <|think|>).

Limitations

Do not regenerate the mmproj with a plain F16 conversion. F16 conversion of v.patch_embd.weight (6912×3840) overflows F16 accumulation during the patch-embedding matmul. Observed outcomes on this exact model:

  • F16 mmproj → repeated <unused49> tokens, or
  • F16 mmproj → silent hallucination (model invents image content instead of reading it)

The F32/BF16 mmproj shipped here is unaffected. This matches upstream llama.cpp issue #24146 (closed not-planned, unfixed), root cause 2: "The patch_embeddings_0 matmul overflows F16 accumulation… forcing F32 accumulation fixes it."

Upstream #24146 root cause 1 (resize not filling the soft-token budget) is also unfixed and was not worked around here; it affects token-count accuracy for images, not correctness of this build's OCR results.

Architecture: Gemma4UnifiedForConditionalGeneration, 48 layers, hidden 3840, FFN 15360, 16 KV heads, sliding-window (1024) + full attention in a 40:8 per-layer pattern (matching the KV cache: 40 SWA layers at 5,120 cells + 8 full layers at 262,144 cells). Native 262,144-token context (gemma4.context_length = 262144). Thinking is enabled by default via the <|think|> token.

Source

  • Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
  • Converter: llama.cpp convert_hf_to_gguf.py, build b1868 (4fea119d)
  • Quantizer: llama-quantize NVFP4 (locally patched build; NVFP4 is not in stock llama.cpp)
  • Chat template: native Gemma 4, taken from the source chat_template.jinja

License

Apache 2.0 (same as the base model).

Run FreedomAISVR/Gemma-4-12B-it-QAT-NVFP4-GGUF with guIDE

Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.

Download guIDE → · Browse 524k+ models · Compare models

Source: Hugging Face · Compare models