---

license: gemma

base_model: google/gemma-4-12b-it

base_model_relation: quantized

pipeline_tag: image-text-to-text

tags:

• gguf
• rocm
• rocmfp4
• amd
• strix-halo
• gfx1151
• gemma4
• llama.cpp

---

Gemma-4-12B-it — ROCmFP4 / ROCmFPX GGUF

AMD-native FP4 / FP8 GGUF builds of google/gemma-4-12b-it, for the ROCmFPX fork of

llama.cpp on RDNA3.5 / Strix Halo (gfx1151). These tensor types do not exist in mainline

llama.cpp — you need a ROCmFPX-capable build to load them.

Multimodal (mmproj included) and shipped with the MTP draft head, which is worth

1.46× here (see below).

Variants — all five in this repo

| file | ftype | size | BPW | token_embd | decode |

|---|---|---|---|---|---|

| gemma-4-12b-it-Q4_0_ROCMFP4_COHERENT.gguf | 102 | 6.50 GiB | 4.68 | q6_K | 26.95 t/s |

| gemma-4-12b-it-Q4_0_ROCMFP4_FAST_COHERENT.gguf | 104 | 6.18 GiB | 4.45 | q6_K | 26.56 t/s |

| gemma-4-12b-it-Q6_0_ROCMFPX_AGENT.gguf | 114 | 10.43 GiB | 7.52 | q8_0 | 17.54 t/s |

| gemma-4-12b-it-Q8_0_ROCMFPX.gguf | 111 | 11.48 GiB | 8.27 | q8_0 | 16.39 t/s |

| gemma-4-12b-it-Q8_0_ROCMFPX_AGENT.gguf | 115 | 11.67 GiB | 8.41 | q8_0 | 15.92 t/s |

Decode measured on an idle Ryzen AI MAX+ 395 (Strix Halo, 128 GB, ROCm 7.2.4), -ngl 999,

-c 4096 -fa on -fit off -np 1, 300-token generations, no draft head — see the MTP

section for the accelerated figures.

⚠️ Tied embeddings — --output-tensor-type is a silent no-op here

gemma-4-12b-it sets tie_word_embeddings: true, so there is no output.weight tensor.

Passing --output-tensor-type does nothing at all; **--token-embedding-type is the only

head protection that applies.** With a 262144-token vocabulary at n_embd 3840 the embedding

is a large share of the file, which is why the 4-bit lands at 4.68 BPW rather than ~4.0.

Measured — not estimated

| quant | correctness | decode median | runs |

|---|---|---|---|

| 102 | 3/3 ✅ | 26.95 t/s | 26.95 / 26.96 / 26.99 / 26.95 / 26.98 / 26.93 / 26.98 / 26.97 / 26.96 / 26.95 / 26.95 / 26.95 |

| 104 | 3/3 ✅ | 26.56 t/s | 26.56 / 26.56 / 26.57 / 26.59 / 26.57 / 26.55 / 26.56 / 26.56 / 26.56 / 26.56 / 26.56 / 26.58 |

| 114 | 3/3 ✅ | 17.54 t/s | 17.54 / 17.54 / 17.54 / 17.54 / 17.55 / 17.54 / 17.54 / 17.55 / 17.54 / 17.54 / 17.54 / 17.54 |

| 111 | 3/3 ✅ | 16.39 t/s * | 16.39 / 15.56 / 16.39 / 16.38 / 16.39 |

| 115 | 3/3 ✅ | 15.92 t/s * | 15.92 / 15.16 / 15.92 / 15.92 / 15.93 |

\* = 5 samples rather than 12. Those two rows each had a single low sample, which a median

absorbs; the 4-bit rows were re-run at 12 samples because two low samples of five moved the

number and inverted the FAST-vs-COHERENT ordering.

Correctness is 17×23 → 391 · capital of Japan → Tokyo · days in 2024 → 366, asserted

against content + reasoning and recorded with finish_reason. This model is a reasoner:

it routinely returns an empty content with the answer in reasoning_content, so a harness

that only reads content will score correct answers as failures.

Size integrity

Every artifact's on-disk size exceeds its dry-run projection by a constant ~%s MiB

header delta (spread across all five: 0.0075 MiB). A varying delta is the signature of a

truncated write; a constant one is just the header.

⭐ Speculative decoding (MTP) — big speedup, with one real caveat

mtp-gemma-4-12b-it-Q8_0.gguf ships in this repo. Unlike the E2B/E4B members of this family,

the draft head is a substantial win here — read the caveat below before enabling it.

| config | decode | vs no drafter | draft acceptance |

|---|---|---|---|

| no drafter | 26.95 t/s | 1.00× | — |

| --spec-draft-n-max 3 | 36.2 t/s | 1.34× | 0.64378 |

| --spec-draft-n-max 5 ← best | 39.25 t/s | 1.46× | 0.57049 |

llama-server -m gemma-4-12b-it-Q4_0_ROCMFP4_COHERENT.gguf \
  --spec-type draft-mtp --model-draft mtp-gemma-4-12b-it-Q8_0.gguf \
  --spec-draft-ngl 999 --spec-draft-n-max 5 \
  -ngl 999 -c 4096 -fa on -fit off -np 1

⭐ Note that the fastest setting is not the one with the highest draft acceptance.

Acceptance measures how often the drafter is right; throughput also pays for the

verification work. Sweep n-max and rank on measured t/s, not on acceptance.

⛔ Known defect: long generations abort the server with the draft head enabled

With --spec-type draft-mtp at -c 4096, a 1500-token generation reliably aborts the

server:

server-context.cpp:395: GGML_ASSERT(spec_i_batch.empty()) failed
  in server_slot::update_batch(llama_batch&)

Reproduced at both --spec-draft-n-max 3 and 5. 300-token generations are unaffected —

we ran 5 per quant here plus 13 consecutive requests in a separate test with no failure, so

the decode figures above are sound. We have not yet isolated whether the trigger is the

single-request length itself or a context shift near the -c limit; **if you enable the

drafter, cap generations conservatively and give yourself context headroom.** Without

--spec-type the model handles long generations normally.

⛔ MTP and vision cannot be used together (upstream llama.cpp PR #20277 — image embeddings

are injected outside the token path and the speculative batch loses its boundary). Run two

configurations: text with the drafter, images with -fa off and no --spec-type.

A note on measurement, because we hit it here

This model showed a reproducible ~8% dip in a minority of decode samples on an otherwise

idle box, while E2B/E4B measured on the same machine and harness were flat. We re-ran at

12 samples per quant to characterise it:

• Q4_0_ROCMFP4_COHERENT [pyfunc] median 26.93 — 26.94 / 26.93 / 26.93 / 26.93 / 26.93 / 26.94 / 26.94 / 26.93 / 26.93 / 26.93 / 26.95 / 26.94
• Q4_0_ROCMFP4_COHERENT [story] median 26.93 — 26.92 / 26.93 / 26.93 / 26.93 / 26.93 / 26.92 / 26.93 / 26.94 / 26.92 / 26.93 / 26.93 / 26.92
• Q4_0_ROCMFP4_FAST_COHERENT [pyfunc] median 26.57 — 26.58 / 26.6 / 26.64 / 26.57 / 26.57 / 26.58 / 26.57 / 26.57 / 26.57 / 26.57 / 26.56 / 26.58

The medians above are taken over 12 samples for this reason. We report every raw sample

rather than a summary so you can see the distribution yourself.

Verification

Each artifact was loaded on real hardware and checked for: exact stat bytes vs the dry-run

projection, actual token_embd type, three correctness answers, and a 5-sample decode median

with two warm-ups discarded. Vision was verified separately with the mmproj on a

four-quadrant colour image.

Credits

Base model google/gemma-4-12b-it. BF16 GGUF source from unsloth/gemma-4-12b-it-GGUF.

FP4/FP8 tensor types from the ROCmFPX fork of llama.cpp.