---

library_name: gguf

license: apache-2.0

tags:

• gemma
• gemma-4
• nvfp4
• gguf

pipeline_tag: text-generation

base_model: google/gemma-4-12B-it

---

Gemma 4 12B Instruct - NVFP4 GGUF

Base Model

This is a quantized version of google/gemma-4-12B-it, Google's instruction-tuned 12 billion parameter language model from the Gemma 4 family.

Gemma 4 12B is a dense, decoder-only transformer trained on a diverse dataset of text and code. It excels at instruction following, reasoning, coding, and multimodal understanding.

About This Quantization

• Format: NVFP4 (NVIDIA FP4) — 4-bit E2M1 values with E4M3 scaling per 16 values
• BPW: 4.68 bits per weight
• File size: 6.5 GB
• Hardware target: NVIDIA Blackwell GPUs (RTX 50 series, SM 120a)
• Note: NVFP4 decode is kernel-limited on consumer GPUs (~50 t/s on RTX 5060 Ti). The FP4 tensor cores primarily accelerate prefill, not decode. MXFP4 variant available for faster decode.

Quantization Details

| Property | Value |

|----------|-------|

| Base model | google/gemma-4-12B-it |

| Architecture | Gemma4ForCausalLM |

| Parameters | 12.08B |

| Layers | 48 |

| Hidden size | 3840 |

| Context length | 128K (tested up to 1M with Q4_0 KV) |

| Quantization | NVFP4 |

| BPW | 4.68 |

| File size | 6.5 GB |

Performance (RTX 5060 Ti 16GB)

| Context | KV Cache | Prompt | Generation |

|---------|----------|--------|------------|

| 128K | Q8_0 | 43.8 t/s | 49.9 t/s |

| 256K | Q8_0 | 244.8 t/s | 49.5 t/s |

| 256K | Q4_0 | 309.8 t/s | 50.8 t/s |

| 512K | Q4_0 | 301.0 t/s | 50.0 t/s |

| 1M | Q4_0 | 47.4 t/s | 50.2 t/s |

| 1M | Q8_0 | 17.4 t/s | 27.8 t/s |

Recommended Settings

llama-cli -m gemma-4-12b-it-nvfp4.gguf -ngl 99 -c 262144 -ctk q8_0 -ctv q8_0 -b 1024 -ub 512 --flash-attn

• -ngl 99 — full GPU offload
• -c 262144 — 256K context (recommended sweet spot)
• -ctk q8_0 -ctv q8_0 — KV cache at Q8_0 for best quality
• -b 1024 -ub 512 — optimized batch sizes
• --flash-attn — essential for long context

Why No MTP?

Gemma 4 models do not include Multi-Token Prediction (MTP) heads. MTP is a Qwen3-specific architecture feature. This is a standard Gemma 4 model with no modifications to the architecture.

Usage with llama.cpp

# Interactive mode
./llama-cli -m gemma-4-12b-it-nvfp4.gguf -ngl 99 -c 262144 -ctk q8_0 -ctv q8_0 --flash-attn -i

# Server mode
./llama-server -m gemma-4-12b-it-nvfp4.gguf -ngl 99 -c 262144 -ctk q8_0 -ctv q8_0 --flash-attn --host 0.0.0.0 --port 8080

License

Apache 2.0 — same as the base model.