---

license: apache-2.0

language:

• en

library_name: gguf

tags:

• gguf
• gemma
• gemma-4
• 12b
• mxfp4
• fp4
• microscaling
• vision
• multimodal

base_model: google/gemma-4-12B-it

pipeline_tag: text-generation

inference: false

quantized_by: freedomaisvr

---

Gemma-4-12B-it-MXFP4-GGUF

MXFP4 (OCP microscaling FP4) quantization of Google's Gemma 4 12B It, a multimodal language model with native vision understanding.

This repository contains two files:

• gemma-4-12b-it-mxfp4.gguf — Text backbone (48 transformer layers, 3840 hidden dim, 262k context) quantized to MXFP4
• mmproj-gemma-4-12b-it-f16.gguf — SigLIP vision encoder + projector at F16 precision (required for image input)

About MXFP4

MXFP4 (Microscaling FP4) is the open-standard 4-bit floating-point format defined by the OCP Microscaling Formats (MX) specification. It uses an E2M1 format (1 sign, 2 exponent, 1 mantissa bit) with per-block shared exponent (scaling factor), providing better dynamic range than INT4 while remaining hardware-agnostic.

| Feature | MXFP4 | Q4_K_M | NVFP4 |

|---|---|---|---|

| Numeric format | E2M1 microscaling FP4 | INT4 block quantization | E4M3 native FP4 |

| Shared exponent (scale) | Per 32 elements | Per 32 elements | None (native FP4) |

| Effective BPW | 4.61 | ~4.50 | 4.68 |

| Hardware support | Any GPU + CPU | Any GPU + CPU | Blackwell only |

| Dynamic range (max normal) | 30 (E2M1) | 7 (INT4, symmetric) | 448 (E4M3) |

| Dequantization overhead | Moderate (scale mul) | Moderate (scale mul) | None |

When to use MXFP4: You want a universal 4-bit format that works on any GPU (NVIDIA, AMD, Intel) or CPU without sacrificing quality. MXFP4 offers better dynamic range than INT4 thanks to its microscaling exponent, making it more robust for outlier-heavy layers.

When to use NVFP4 instead: You have a Blackwell GPU (RTX 50-series) — the native FP4 tensor cores give higher throughput.

When to use Q4_K_M: You need maximum compatibility with older llama.cpp versions (pre-June 2025) or CPU-only inference.

Files

| Filename | Type | Size | BPW | Description |

|---|---|---|---|---|

| gemma-4-12b-it-mxfp4.gguf | MXFP4 quantized (text) | 6.87 GB | 4.61 | 48-layer text backbone with hybrid attention |

| mmproj-gemma-4-12b-it-f16.gguf | Vision encoder (F16) | 122 MB | 16.0 | SigLIP vision embedder + GEMMA4UV projector |

Quantization Characteristics

| Metric | Value |

|---|---|

| Input format | F16 GGUF (23.83 GB, 667 tensors) |

| Output format | MXFP4 GGUF (6.87 GB, 667 tensors) |

| Quantization type | LLAMA_FTYPE_MOSTLY_MXFP4 (type 41) |

| Compression ratio | 3.47× (23.83 GB → 6.87 GB) |

| Quantization time | ~255 seconds on RTX 5060 Ti |

| 1D tensors (norms, scales) | Kept at Q8_0 |

| Attention + FFN weights | Converted to MXFP4 |

Model Description

Gemma 4 12B is part of Google's fourth-generation Gemma family, featuring:

• 48 transformer layers with 3840 hidden dimensions and 15360 FFN intermediate size
• Hybrid attention: 40 sliding-window layers (window 1024, kv_heads=8, head_dim=256) interleaved with 8 full-attention layers (kv_heads=2, head_dim=512) in a 5:1 pattern
• Context window: up to 262,144 tokens
• RoPE scaling: separate frequency bases for sliding window and full attention
• Final logit softcapping: stabilizes large-vocabulary predictions
• Vision: SigLIP-based lightweight patch embedder with learned positional encoding
• Instruction-tuned: optimized for chat and instruction-following

The base model is google/gemma-4-12B-it under Apache 2.0 license.

Usage

llama.cpp CLI

# Text-only inference
./llama-cli \
  -m gemma-4-12b-it-mxfp4.gguf \
  -p "Explain quantum computing in simple terms" \
  -n 512

# Vision inference (requires mmproj)
./llama-cli \
  -m gemma-4-12b-it-mxfp4.gguf \
  --mmproj mmproj-gemma-4-12b-it-f16.gguf \
  --image diagram.png \
  -p "Explain what this diagram shows" \
  -n 512

llama.cpp Server (OpenAI-compatible API)

./llama-server \
  -m gemma-4-12b-it-mxfp4.gguf \
  --mmproj mmproj-gemma-4-12b-it-f16.gguf \
  --port 8080 \
  -ngl 99 \
  -c 8192

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="gemma-4-12b-it-mxfp4.gguf",
    mmproj="mmproj-gemma-4-12b-it-f16.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,
)

output = llm("What is the capital of France?", max_tokens=128)
print(output["choices"][0]["text"])

Download

from huggingface_hub import hf_hub_download

model_path = hf_hub_download(
    repo_id="FreedomAISVR/Gemma-4-12B-it-MXFP4-GGUF",
    filename="gemma-4-12b-it-mxfp4.gguf"
)
mmproj_path = hf_hub_download(
    repo_id="FreedomAISVR/Gemma-4-12B-it-MXFP4-GGUF",
    filename="mmproj-gemma-4-12b-it-f16.gguf"
)

Thinking / Reasoning Behavior

Gemma 4 supports structured reasoning using <|channel>thought tags. The chat template handles reasoning as follows:

• enable_thinking=false (default): The template inserts empty think tags (<|channel>thought\n<channel|>) at the start of the model's generation turn, suppressing thinking.
• enable_thinking=true: The model may generate internal reasoning tokens enclosed in <|channel>thought...<channel|>.

In LM Studio, reasoning sections are rendered as collapsible blocks with proper reasoning.parsing configuration.

Conversion Pipeline

google/gemma-4-12B-it (23.92 GB, safetensors)
  │
  ├─ convert_hf_to_gguf.py --outtype f16 (llama.cpp d403f00)
  │     → gemma-4-12b-f16.gguf (23.83 GB, 667 tensors, text backbone only)
  │
  ├─ convert_hf_to_gguf.py --mmproj --outtype f16
  │     → mmproj-gemma-4-12b-f16.gguf (122 MB, 11 tensors)
  │
  └─ llama-quantize.exe MXFP4
        → gemma-4-12b-it-mxfp4.gguf (6.87 GB, 667 tensors, 4.61 BPW)

Built with llama.cpp commit d403f00. Google's original chat template (17,466 bytes) is preserved as-is.

Hardware Requirements

| GPU | VRAM | Model + KV Cache (4k ctx) | Estimated Speed |

|---|---|---|---|

| RTX 5060 Ti 16GB | 16 GB | ~10 GB | ~45 tok/s |

| RTX 4090 24GB | 24 GB | ~10 GB | ~55 tok/s |

| RX 7900 XTX 24GB | 24 GB | ~10 GB | ~35 tok/s |

| CPU (6+ cores) | System RAM | ~7 GB | ~5-8 tok/s |

Minimum VRAM: ~10 GB for the model (~7 GB at 4.61 BPW + ~2.5 GB KV cache at 8k ctx).

Hardware Compatibility

| Backend | MXFP4 Support | Notes |

|---|---|---|

| CUDA (any SM) | ✅ Universal | Works on all NVIDIA GPUs |

| CPU (llamafile) | ✅ Universal | Requires recent llama.cpp |

| Vulkan | ✅ Universal | Tested on AMD + NVIDIA |

| Metal (Apple) | ⚠️ Experimental | Limited testing |

| AMD ROCm | ✅ Universal | Tested on RX 7900 series |

License

Apache 2.0, as per Google's Gemma 4 license.