steampunque/gemma-3-12b-it-MP-GGUF overview

---

license: gemma

base_model: google/gemma-3-12b-it

base_model_relation: quantized

tags:

• gemma
• gemma-3
• GGUF
• quantized
• 4-bit

---

Mixed Precision GGUF layer quantization of gemma-3-12b-it by Google

Original model: https://huggingface.co/google/gemma-3-12b-it

The hybrid quant employs different quantization levels on a per layer basis to increased

flexibility of trading off performance vs file size. Less parameter bits are used at deep layers

and more bits at cortex layers to simultaneously optimize quantized size and model performance.

An extended layer definition E quant Q4_E_H for the model is defined as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q6_K_666"],[1 ,"Q6_K_655"],[2 ,"Q5_K_655"],[3 ,"Q5_K_555"],[4 ,"Q4_K_544"],[5 ,"Q4_K_666"],[6 ,"Q4_K_544"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_544"],[9 ,"Q4_K_544"],[10,"Q4_K_544"],[11,"Q4_K_666"],[12,"Q4_K_544"],[13,"Q4_K_544"],[14,"Q4_K_544"],[15,"Q4_K_544"],
   [16,"Q4_K_544"],[17,"Q4_K_555"],[18,"Q4_K_544"],[19,"Q4_K_544"],[20,"Q4_K_544"],[21,"Q4_K_544"],[22,"Q4_K_544"],[23,"Q4_K_555"],
   [24,"Q4_K_644"],[25,"Q4_K_644"],[26,"Q4_K_644"],[27,"Q4_K_644"],[28,"Q4_K_644"],[29,"Q4_K_655"],[30,"Q4_K_644"],[31,"Q4_K_644"],
   [32,"Q4_K_654"],[33,"Q4_K_654"],[34,"Q4_K_654"],[35,"Q4_K_655"],[36,"Q4_K_654"],[37,"Q4_K_654"],[38,"Q4_K_654"],[39,"Q4_K_654"],
   [40,"Q4_K_654"],[41,"Q5_K_666"],[42,"Q4_K_665"],[43,"Q4_K_666"],[44,"Q5_K_666"],[45,"Q5_K_666"],[46,"Q5_K_668"],[47,"Q6_K_886"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The quant was optimized for high reasoning + knowledge performance across a range of test prompts while sized to approximate

Q4_K_M bpw.

Comparison:

Quant | size | PPL | Comment

---------|---------|------|-----------

Q4_K_M | 7.3e9 | - | default embed and output

Q4_E_H | 7.5e9 | 9.1 | Q6_K embed Q6_K output

Usage:

gemma-3 12b is a vision capable model. It can be used together with its multimedia projector layers to process images

and text inputs and generate text outputs. The mmproj file is made available in this repository. To test vision mode follow

the docs in the mtmd readme in the tools directory of the source tree https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/README.md .

The model also uses sliding window attention. Use of llama.cpp b5554 and above is recommend for support of the SWA mode.

If --swa-full flag is used, the old method of keeping all KV memory and masking out everything outside the SWA window is used.

Running:

The model can be speculated with gemma 3 270m it. Approximate performance using a downstream llama.cpp server with cuda backend and custome speculation

code on a RTX 4070 (b9940 downstream):

ND | QKV | NKV | PP | TG

---|-----|---------|-----|------

4 | F16 | 50k | 1000 | 86

0 | F16 | 60k | 1000 | 53

4 | Q8_0 | 82k | 1000 | 82

0 | Q8_0 | 95k | 1000 | 52

The model passes a basic long context test but will not correctly solve the long runescape prompt https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt

trimmed back to fit in available context. This prompt was most likely not available during pretrain of gemma 3.

Vision:

The model correctly identifies two tough bird-id images which gemma-4-12b-it fails, although gemma-4 does score higher on synthetic vision evals.

Code:

The model is capable of creating working code on simple prompts but failed about half the test cases given.

Benchmarks:

Benchmarks for the model (vision only) are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

| Link | Type | Size/e9 B | Notes |

|------|------|-----------|-------|

| gemma-3-12b-it.Q4_E_H.gguf | Q4_E_H | 7.5e9 B | ~Q4_K_M size |

| gemma-3-12b-it.mmproj.gguf | mmproj | 0.85e9 B | multimedia projector |

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040