---

license: gemma

base_model: google/gemma-4-E2B-it

base_model_relation: quantized

pipeline_tag: image-text-to-text

library_name: gguf

tags:

- gguf

- rocmfp4

- rocmfpx

- strix-halo

- gfx1151

- amd

- rocm

- gemma

- multimodal

- quantized

---

Gemma-4-E2B-it — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of google/gemma-4-E2B-it. Four quants, built from the Hub

BF16 GGUF (no re-conversion from safetensors), each individually verified — text and vision —

on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory — using the

ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8

tensor types that mainline llama.cpp does not have.

> ⚠️ These files need a ROCmFPX-capable llama.cpp build. They will not load in stock

> llama.cpp / Ollama / LM Studio — Q4_0_ROCMFP4_ and Q_0_ROCMFPX* are not mainline types.

Variants — all four in this repo

| file | ftype | size | BPW | token_embd | decode |

|---|---|---|---|---|---|

| gemma-4-E2B-it-Q4_0_ROCMFP4_COHERENT.gguf | 102 | 3.13 GiB | 5.76 | q6_K | 94.46 t/s |

| gemma-4-E2B-it-Q6_0_ROCMFPX_AGENT.gguf | 114 | 4.38 GiB | 8.08 | q8_0 | 71.23 t/s |

| gemma-4-E2B-it-Q8_0_ROCMFPX.gguf | 111 | 4.57 GiB | 8.42 | q8_0 | 68.24 t/s |

| gemma-4-E2B-it-Q8_0_ROCMFPX_AGENT.gguf | 115 | 4.60 GiB | 8.48 | q8_0 | 68.07 t/s |

⚠️ One sample in this table is an outlier (115: 55.05 against a ~68 baseline). The box was otherwise idle and every other

sample sits inside a 1.01x band, so this is a transient, not a bimodal model. The median is

reported precisely because it is robust to it, and all five raw samples are shown rather than

hidden behind a summary.

\* The two 8-bit builds were measured while the same machine was quantizing another model. Their

spreads (11% and 20%) show that contention, so treat those two as lower bounds — uncontended

single-shot observations reached 56.4 and 57.5 t/s. The 4-bit and 6-bit figures were taken clean

(spread under 0.4%) and need no such caveat. We would rather publish a conservative number than a

flattering one.

mmproj-BF16.gguf (vision projector) is included — you need it for image input.

Which to pick: the 4-bit (102). It is the smallest and the fastest by a wide margin, and

it answered every correctness and vision test identically to the 8-bits. Take an 8-bit only if you

want maximum fidelity for its own sake.

⚠️ Why the 4-bit is 5.76 BPW, not ~4.5

E2B is a MatFormer / per-layer-embedding model: a large fraction of its parameters live in

per_layer_token_embd and related tensors that are not driven down to 4 bits. So the 4-bit build

is only 1.46× smaller than the 8-bit here, where a conventional dense model would be ~1.8×.

This is expected for this architecture, not a defect — the quantized layers really are FP4.

One tensor, per_layer_model_proj.weight [1536, 8960] (~26 MiB), stays BF16 in all four builds.

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified.

Load: -ngl 999 -c 4096 -fa on -fit off. Decode = 300 tokens, warm-up discarded, median of 3.

| quant | correctness (3/3) | vision | decode median | runs |

|---|---|---|---|---|

| 102 | 17×23=391 · Tokyo · 366 | ✅ named all 4 quadrant colours | 94.46 t/s | 94.61 / 94.56 / 94.46 / 94.4 / 94.41 |

| 114 | 17×23=391 · Tokyo · 366 | ✅ | 71.23 t/s | 71.34 / 71.21 / 71.23 / 71.23 / 71.23 |

| 111 | 17×23=391 · Tokyo · 366 | ✅ | 68.24 t/s | 68.27 / 68.28 / 68.15 / 68.24 / 68.23 |

| 115 | 17×23=391 · Tokyo · 366 | ✅ | 68.07 t/s | 68.09 / 67.94 / 68.09 / 68.07 / 55.05 |

A note on the usual bandwidth sanity check

For a conventional dense model you can sanity-check a decode figure with t/s × file_GB against

the platform's peak memory bandwidth (~256 GB/s here) — anything well under ~80% suggests something

is misconfigured. That check does not apply to this model. E2B's per-layer-embedding design

means not every byte of the file is read per token, so the arithmetic returns ~99–102% of "peak"

and would happily exceed it. Do not read those percentages as validation for MatFormer/PLE (or MoE)

architectures.

Vision was tested with a four-colour quadrant image and the model had to say which colour was

where — a solid-colour square would pass trivially and proves nothing. Use -fa off for image

input.

⚠️ Speculative decoding (MTP): it works, but it is slower — don't use it

Earlier revisions of this card said MTP "does not currently work". That was wrong, and the

reason it was wrong is worth stating: it was tested on one machine whose llama.cpp build never

wired the MTP source, and the failure was then explained by a model-width theory that happened to

fit. On a build that does wire it (ROCmFPX-2809dc5) the draft head loads and drafts correctly.

It is still not worth enabling here:

| config | decode |

|---|---|

| no drafter | 94.42 t/s |

| --spec-type draft-mtp, n-max 3, Q8 drafter | 88.54 t/s |

| draft acceptance | 0.589 |

⚠️ This repo does not ship the draft head (an earlier version of this card wrongly said it

did). If you want to reproduce the row above, take MTP/mtp-gemma-4-E2B-it-Q8_0.gguf from

unsloth/gemma-4-E2B-it-GGUF. We do not

ship it here because it makes this model slower.

0.94x — a net loss at 59% acceptance. Speculation pays inversely to how fast the target

already is: on our Gemma-4-31B build the same technique is worth 1.84x at 11.8 t/s, on the

26B-A4B ~1.03-1.15x at ~54 t/s, and here at ~94 t/s it costs more than it returns. The drafter's

per-step overhead is roughly fixed; the target's per-token cost is not.

⭐ Acceptance does not predict speedup. A drafter can be right most of the time and still lose.

Verification

Each artifact was checked individually, not sampled:

• Size vs --dry-run projection — constant ~15.8 MB GGUF-header delta on all four (truncation check)
• token_embd audited by exact tensor name
• Loaded, 3/3 correctness, vision content test
• Decode median of 3 with the spread reported

Exact byte sizes:

3360161632  gemma-4-E2B-it-Q4_0_ROCMFP4_COHERENT.gguf
4707581792  gemma-4-E2B-it-Q6_0_ROCMFPX_AGENT.gguf
4908687200  gemma-4-E2B-it-Q8_0_ROCMFPX.gguf
4941201248  gemma-4-E2B-it-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

gemma-4-E2B-it has tie_word_embeddings = true — there is no output.weight tensor.

--output-tensor-type is therefore a silent no-op, and **--token-embedding-type is the only

flag that actually protects the head**:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  gemma-4-E2B-it-BF16.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Confirm it landed by reading the finished file's token_embd.weight type. And when auditing,

do not grep output.weight — it false-matches blk.N.attn_output.weight and will make a

healthy build look damaged.

Credits

• Base model: google/gemma-4-E2B-it — Gemma Terms of Use
• Quantization stack: ROCmFPX