---

license: gemma

base_model: google/gemma-4-E4B-it

base_model_relation: quantized

pipeline_tag: image-text-to-text

library_name: gguf

tags:

- gguf

- rocmfp4

- rocmfpx

- strix-halo

- gfx1151

- amd

- rocm

- gemma

- multimodal

- quantized

---

Gemma-4-E4B-it — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of google/gemma-4-E4B-it. Four quants, built from the Hub

BF16 GGUF (no re-conversion from safetensors), each individually verified — text and vision —

on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory — using the

ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8

tensor types that mainline llama.cpp does not have.

> ⚠️ These files need a ROCmFPX-capable llama.cpp build. They will not load in stock

> llama.cpp / Ollama / LM Studio — Q4_0_ROCMFP4_ and Q_0_ROCMFPX* are not mainline types.

Variants — all four in this repo

| file | ftype | size | BPW | token_embd | decode |

|---|---|---|---|---|---|

| gemma-4-E4B-it-Q4_0_ROCMFP4_COHERENT.gguf | 102 | 4.83 GiB | 5.19 | q6_K | 56.15 t/s |

| gemma-4-E4B-it-Q6_0_ROCMFPX_AGENT.gguf | 114 | 6.96 GiB | 7.48 | q8_0 | 39.04 t/s |

| gemma-4-E4B-it-Q8_0_ROCMFPX.gguf | 111 | 7.36 GiB | 7.91 | q8_0 | 37.62 t/s |

| gemma-4-E4B-it-Q8_0_ROCMFPX_AGENT.gguf | 115 | 7.43 GiB | 7.98 | q8_0 | 37.26 t/s |

⚠️ One sample in this table is an outlier (102: 47.21 against a ~56 baseline). The box was otherwise idle and every other

sample sits inside a 1.01x band, so this is a transient, not a bimodal model. The median is

reported precisely because it is robust to it, and all five raw samples are shown rather than

hidden behind a summary.

mmproj-BF16.gguf (vision projector) is included — you need it for image input.

Which to pick: the 4-bit (102). It is the smallest and the fastest by a wide margin, and

it answered every correctness and vision test identically to the 8-bits. Take an 8-bit only if you

want maximum fidelity for its own sake.

⚠️ Why the 4-bit is 5.76 BPW, not ~4.5

E4B is a MatFormer / per-layer-embedding model: a large fraction of its parameters live in

per_layer_token_embd and related tensors that are not driven down to 4 bits. So the 4-bit build

is only 1.53× smaller than the 8-bit here, where a conventional dense model would be ~1.8×.

This is expected for this architecture, not a defect — the quantized layers really are FP4.

One tensor, per_layer_model_proj.weight stays BF16 in all four builds. 720 tensors per file.

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified.

Load: -ngl 999 -c 4096 -fa on -fit off. Decode = 300 tokens, warm-up discarded, median of 3.

| quant | correctness (3/3) | vision | decode median | runs | spread |

|---|---|---|---|---|---|

| 102 | 17×23=391 · Tokyo · 366 | ✅ named all 4 quadrant colours | 56.15 t/s | 56.55 / 56.15 / 47.21 / 56.15 / 56.02 | 1.198× |

| 114 | 17×23=391 · Tokyo · 366 | ✅ | 39.04 t/s | 39.25 / 39.12 / 39.04 / 38.99 / 38.94 | 1.008× |

| 111 | 17×23=391 · Tokyo · 366 | ✅ | 37.62 t/s | 37.65 / 37.62 / 37.62 / 37.62 / 37.6 | 1.001× |

| 115 | 17×23=391 · Tokyo · 366 | ✅ | 37.26 t/s | 37.26 / 37.26 / 37.25 / 37.22 / 37.26 | 1.001× |

All spreads ≤1.03×, measured with nothing else building on the box — these are clean numbers.

A note on the usual bandwidth sanity check

For a conventional dense model you can sanity-check a decode figure with t/s × file_GB against

the platform's peak memory bandwidth (~256 GB/s here) — anything well under ~80% suggests something

is misconfigured. That check does not apply to this model. E4B's per-layer-embedding design

means not every byte of the file is read per token, so the arithmetic returns well over 100% of "peak"

and would happily exceed it. Do not read those percentages as validation for MatFormer/PLE (or MoE)

architectures.

Vision was tested with a four-colour quadrant image and the model had to say which colour was

where — a solid-colour square would pass trivially and proves nothing. Use -fa off for image

input.

⚠️ Speculative decoding (MTP): the draft head loads, but the server aborts — don't use it

Earlier revisions of this card said MTP "does not currently work" and blamed a shared-KV /

model-width problem. That explanation was wrong. The draft head initialises correctly

(common_speculative_state_draft_mtp: n_embd=2560, backend_sampling=1). What actually happens:

| config | result |

|---|---|

| -fa on + --spec-type draft-mtp | server aborts in ggml_cuda_flash_attn_ext_tile_case<512, 512> (fattn-tile.cuh:1316) |

| -fa off + --spec-type draft-mtp | loads and drafts (acceptance 0.616) but we could not obtain a stable benchmark |

⚠️ This repo does not ship the draft head (an earlier version of this card wrongly said it

did). It lives in

unsloth/gemma-4-E4B-it-GGUF under MTP/.

The -fa on abort is not a speculation bug at all — it is an uncompiled flash-attention tile

case in the HIP backend, and the speculative path is simply the code path that reaches it.

Every decode number above is measured without a drafter, which is the configuration we

recommend for this model today.

Verification

Each artifact was checked individually, not sampled:

• Size vs --dry-run projection — constant ~15.82 MB GGUF-header delta on all four (spread 5.9 KB) (truncation check)
• token_embd audited by exact tensor name
• Loaded, 3/3 correctness, vision content test
• Decode median of 3 with the spread reported

Exact byte sizes:

5185932160  gemma-4-E4B-it-Q4_0_ROCMFP4_COHERENT.gguf
7472712576  gemma-4-E4B-it-Q6_0_ROCMFPX_AGENT.gguf
7906233216  gemma-4-E4B-it-Q8_0_ROCMFPX.gguf
7974636416  gemma-4-E4B-it-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

gemma-4-E4B-it has tie_word_embeddings = true — there is no output.weight tensor.

--output-tensor-type is therefore a silent no-op, and **--token-embedding-type is the only

flag that actually protects the head**:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  gemma-4-E4B-it-BF16.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Confirm it landed by reading the finished file's token_embd.weight type. And when auditing,

do not grep output.weight — it false-matches blk.N.attn_output.weight and will make a

healthy build look damaged.

Credits

• Base model: google/gemma-4-E4B-it — Gemma Terms of Use
• Quantization stack: ROCmFPX