---

license: gemma

base_model: google/gemma-4-31B-it

base_model_relation: quantized

pipeline_tag: image-text-to-text

library_name: gguf

tags:

- gguf

- rocmfp4

- rocmfpx

- strix-halo

- gfx1151

- amd

- rocm

- gemma

- multimodal

- quantized

---

Gemma-4-31B-it — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of google/gemma-4-31B-it. Four quants, built from the Hub

BF16 GGUF (no re-conversion from safetensors), each individually verified — text and vision —

on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory, ROCm 7.2.4 —

using the ROCmFPX llama.cpp fork, which adds AMD-native

FP4/FP8 tensor types that mainline llama.cpp does not have.

> ⚠️ These files need a ROCmFPX-capable llama.cpp build. They will not load in stock

> llama.cpp / Ollama / LM Studio — Q4_0_ROCMFP4_ and Q_0_ROCMFPX* are not mainline types.

Variants — all four in this repo

| file | ftype | size | BPW | token_embd | decode + MTP | decode alone | speedup |

|---|---|---|---|---|---|---|---|

| gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf | 102 | 16.44 GiB | 4.60 | q6_K | 21.67 t/s | 11.77 | 1.84× |

| gemma-4-31B-it-Q6_0_ROCMFPX_AGENT.gguf | 114 | 26.63 GiB | 7.45 | q8_0 | 14.82 t/s | 7.33 | 2.02× |

| gemma-4-31B-it-Q8_0_ROCMFPX.gguf | 111 | 29.54 GiB | 8.26 | q8_0 | 14.86 t/s | 6.71 | 2.21× |

| gemma-4-31B-it-Q8_0_ROCMFPX_AGENT.gguf | 115 | 30.04 GiB | 8.40 | q8_0 | 15.34 t/s | 6.57 | 2.33× |

mmproj-BF16.gguf (vision projector) is included — required for image input.

Which to pick: the 4-bit (102) — 21.67 t/s with the drafter, 1.46× the fastest 8-bit at

55% of the size, with identical correctness and vision results.

If you want an 8-bit, take the AGENT variant Q8_0_ROCMFPX_AGENT (115), not the plain one:

with speculation on it is both faster (15.34 vs 14.86) and accepts more drafts (0.619 vs 0.603),

because the AGENT recipe holds more attention-output tensors at higher precision — which is exactly

what raises draft acceptance. (Without a drafter the ordering reverses and plain Q8 wins; the AGENT

recipe only pays when you are actually speculating.)

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified, ROCm 7.2.4.

Load: -ngl 999 -c 4096 -fa on -fit off. Decode = 300 tokens, warm-up discarded, median of 3.

With the drafter (n-max 5), correctness re-checked under the shipping config:

| quant | run 1 / 2 / 3 | median | acceptance | correctness |

|---|---|---|---|---|

| 102 | 21.69 / 21.68 / 21.66 | 21.68 | 0.735 | 391 · Tokyo · 366 ✅ |

| 114 | 14.82 / 14.79 / 14.86 | 14.82 | 0.629 | ✅ |

| 111 | 14.80 / 14.86 / 14.97 | 14.86 | 0.603 | ✅ |

| 115 | 15.25 / 15.34 / 15.39 | 15.34 | 0.619 | ✅ |

Without the drafter (vision path, -fa off):

| quant | run 1 / 2 / 3 | median | spread | vision (4-colour quadrants) |

|---|---|---|---|---|

| 102 | 11.81 / 11.81 / 11.81 | 11.81 | 1.000 | ✅ TL red · TR blue · BL yellow · BR green |

| 114 | 7.33 / 7.33 / 7.33 | 7.33 | 1.000 | ✅ same |

| 111 | 6.72 / 6.71 / 6.71 | 6.71 | 1.001 | ✅ same |

| 115 | 6.59 / 6.57 / 6.57 | 6.57 | 1.003 | ✅ same |

Spreads ≤1.003 with nothing else running — clean numbers, not contended ones. Speculative decoding

is output-preserving (the target verifies every drafted token) and the correctness answers are

identical with and without it.

Why you can trust the no-drafter numbers

This model is conventionally dense, so weights stream once per token and t/s × file_GB gives

effective memory bandwidth. Against Strix Halo's ~256 GB/s peak, all four land at 81–83% — the

hardware ceiling, so nothing is spilling to CPU or silently misconfigured. (With MTP the model

verifies several tokens per weight read, which is exactly how it exceeds that ceiling.)

Vision was tested with a 512² four-colour quadrant image, asking which colour is where. A

solid-colour square would pass trivially and proves nothing. Use -fa off for image input.

⚠️ Give vision prompts max_tokens: 1024; at 256 the model spends the budget thinking and returns

finish_reason: length with empty content, which looks like a failure and is not.

⚡ Speculative decoding (MTP) — supported, and worth 1.84×

This model has an official drafter, google/gemma-4-31B-it-assistant. **It is included in this

repo** as mtp-gemma-4-31B-it-Q8_0.gguf (491 MB) so the numbers above are reproducible.

Recommended text command — this is what produced 21.67 t/s:

llama-server -m gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf \
  --spec-type draft-mtp --model-draft mtp-gemma-4-31B-it-Q8_0.gguf \
  --spec-draft-ngl 999 --spec-draft-n-max 5 \
  -ngl 999 -c 4096 -fa on -fit off

Measured tuning (median of 3 each, code prompt, idle box) — don't guess these:

| setting | result |

|---|---|

| n-max 5 | 21.67 t/s ← best |

| n-max 3 | 21.19 |

| n-max 4 | 20.81 |

| n-max 6 | 20.65 |

| n-max 8 | 12.83 — collapses |

| Q8 drafter | 20.81 @ n4 ← use this |

| BF16 drafter | 18.84 @ n4 — slower despite higher acceptance (0.704 vs 0.688); the bigger drafter costs more than it returns |

| p_min 0.5 / 0.0 | no reliable gain — apparent +0.9% carries 7× the run-to-run spread |

Draft acceptance at the recommended config is 0.649, mean accepted length 3.18.

Acceptance is workload-dependent: code drafts accept far better than prose, so treat 21.67 as a

code-workload figure and expect less on free-form text.

⛔ MTP and vision cannot be used together

Enabling the drafter and passing an image crashes the server:

process: missing MTP boundary for seq_id=2 pos=132
srv update_slots: failed to process speculative batch
server-context.cpp:3202: fatal error

Image embeddings are injected outside the normal token path and the speculative batch loses its

boundary (upstream llama.cpp PR #20277). **For vision, drop the --spec-type flags and use

-fa off** — that path is fully verified (see below) and runs at 11.81 t/s.

Verification

Each artifact was checked individually:

• Size vs --dry-run projection — constant ~15.3 MiB GGUF-header delta across all four
• token_embd audited by exact tensor name (not substring — output.weight false-matches

blk.N.attn_output.weight)

• Loaded, 3/3 correctness, real-content vision test, decode median of 3 with spread reported

  514687104  mtp-gemma-4-31B-it-Q8_0.gguf   (drafter — needed for the headline speed)
17651002048  gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf
28594301632  gemma-4-31B-it-Q6_0_ROCMFPX_AGENT.gguf
31720467136  gemma-4-31B-it-Q8_0_ROCMFPX.gguf
32252046016  gemma-4-31B-it-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

gemma-4-31B-it has tie_word_embeddings = true — there is no output.weight tensor.

--output-tensor-type is a silent no-op; **--token-embedding-type is the only flag that

actually protects the head**:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  gemma-4-31B-it-BF16.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Verify by reading the finished file's token_embd.weight type — do not assume it worked because

the flag was accepted.

Credits

• Base model: google/gemma-4-31B-it — Gemma Terms of Use
• Quantization stack: ROCmFPX