---

license: apache-2.0

base_model: google/gemma-4-E2B-it

tags:

• gguf
• gemma4
• e-series
• per-layer-embeddings
• quantized
• xyntetik-runner

pipeline_tag: text-generation

---

gemma-4-E2B-it, Q4_0/Q4_K mixed (GGUF)

<!-- BEGIN runner-toolcalling -->

Serve with xyntetik-runner &mdash; tool calls that still parse when the budget runs out

This GGUF is built to be served with xyntetik-runner, a single-binary, dependency-free C11 inference engine (CPU / CUDA / Metal, OpenAI-compatible server). Its one differentiator you cannot get from another engine:

> When a tool call runs past its token budget, Runner closes it to the smallest schema-legal document, so the arguments still parse.

Most OpenAI-compatible engines return finish_reason: "length" with an empty or malformed tool_calls the caller has to repair or retry. Runner does forced-truncation recovery: once a JSON document starts, it emits a legal ending when the budget expires. On local models &mdash; tight context, slow generation &mdash; that is the difference between an agent loop that finishes and one that retries from scratch.

Measured &mdash; what each engine hands the caller when the token budget cuts a tool call short (same box, same tool schema, same prompt, tool_choice:"required", temperature 0, budgets 1&rarr;64):

| engine | budget too small (1&ndash;16 tokens) | enough budget (64, control) |

|---|---|---|

| Runner | executable tool_calls, arguments parse | completes |

| vLLM 0.27.1 | no call; protocol framing leaks into content | completes |

| llama.cpp b10488 | no call; leak, then tool_calls with unparseable args | completes |

| Ollama 0.32.14 | no call; empty content, then HTTP 500 | completes |

| TensorRT-LLM 1.2.1 &dagger; | no call; <tool_call> leak, then empty content | completes |

| SGLang 0.5.17 &dagger; | no call; <tool_call> leak, then empty content | completes |

Every engine completes at 64 tokens &mdash; the failure is truncation, not misconfiguration. Below that, only Runner returns an executable call. This is the behaviour across every OpenAI-compatible engine measured, not a claim about engines not measured. &dagger; TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute; truncation recovery is a property of the runtime, not the model.

Constrained decoding also holds the shape of a tool call under quantization: on a full quant ladder, schema conformance and tool selection stayed 100% down to Q4_0 (argument agreement decayed to 50%) &mdash; it guarantees the shape of a call at any quantization, not its contents.

Quickstart

# build the single binary (no deps), then serve this GGUF:
runner -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf --serve
# point any OpenAI-compatible client at http://localhost:8080/v1

Full recipe, raw responses, and the per-release regression gate: the truncation benchmark.

<!-- END runner-toolcalling -->

Run it in 60 seconds

No Python, no dependencies, the engine is a single 600 KB binary:

# Apple Silicon Mac
curl -L https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/runner-macos-arm64.tar.gz | tar xz
curl -L -o gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf \
  https://huggingface.co/Joakimpalm-Zen/gemma-4-E2B-it-Q4_0-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf
./runner-macos-arm64 -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf -i

2.63 GB on disk, comfortable on an 8 GB Mac. Linux and Windows builds are on

the same release page.

Serve it on a loopback OpenAI-compatible endpoint instead:

./runner-macos-arm64 -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf --serve

The engine is Xyntetik Runner,

a from-scratch C inference engine in public alpha. If you run this,

./runner-macos-arm64 --caps prints one JSON document describing your machine

and what it admits, pasting that into an issue is the single most useful thing

you can send us, and it takes ten seconds.

What this file actually is, corrected 2026-08-11

This card previously called the file "a straight Q4_0 conversion". **That was

wrong, and an external evaluation caught it.** The true content, from the

tensor table of the file itself:

F32: 283   Q4_K: 194   Q4_0: 124        general.file_type = 15 (Q4_K_M)

It was produced by requantizing a Q4_K_M GGUF conversion of

google/gemma-4-E2B-it with runner --quantize --quant q4_0. The quantizer

has a never-grow retention rule, and Q4_K and Q4_0 are both 4.5 bits per

weight, so every Q4_K tensor was silently kept at the higher-quality K

format, and only the parent's Q5/Q6/Q8 tensors actually moved to Q4_0. The

result is slightly BETTER than a true Q4_0 at the same size, but it is a

mixed file, and both the old filename and the old card said otherwise.

The same class of mislabeling is what the runner's own certification matrix

calls the "mixed-tensor trap" when third-party quantizations do it. The same

standard applies here: the file is renamed to say what it is, the bytes are

UNCHANGED (hash below), and the quantizer defect that produced the stale

general.file_type metadata is fixed upstream (runner commit 9a0054e:

the declared type is now derived from the output histogram and the histogram

is printed at quantize time).

• File: gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf (2,631,015,904 bytes)
• SHA256: 16ffaa2ea568e7a0c9b5c6ddbd39a376879f9ac007861334974de39f4cc52e3f
• Source: google/gemma-4-E2B-it via a Q4_K_M GGUF conversion
• Built with: runner --quantize --quant q4_0 (pre-9a0054e metadata behavior)

> Should you use this file? Often, no, and that is worth saying plainly.

> Google publishes official Gemma-4 GGUFs, they are ungated and Apache 2.0,

> and for ordinary use you should prefer them. This artifact exists so that

> benchmarks quoting "gemma-4-E2B at 2.63 GB" can be reproduced against the

> exact bytes they were measured on, hash included. That is its whole job -

> which is also why the rename kept the bytes and the hash identical.

What the E-series actually is

E2B is not "Gemma 4, small". It is a distinct architecture, and two of its

properties break runtimes that assume otherwise:

Per-layer embeddings (PLE). Every layer folds its own embedding table into

the residual, rather than sharing one input embedding. Plus a tail of layers

that own no KV of their own (shared-KV geometry). A runtime that implements

"gemma-4" as a dense transformer will load this file and produce nonsense.

Per-layer FFN widths. gemma4.feed_forward_length is an **array of 35

values**, 6144 for the first 15 layers, 12288 for the remaining 20, because

E2B sets use_double_wide_mlp. This is the interesting compatibility trap:

gemma4.feed_forward_length = [6144 x15, 12288 x20]     # not a scalar

A loader that reads that key with a scalar getter gets 0 and reports

"missing model hyperparameters". Xyntetik Runner hit exactly that, and

gemma-4-E4B masked it for weeks because E4B's checkpoint does not set the

flag, its key is a plain scalar, so it loads fine and the missing path is

never exercised. If your runtime loads E4B and fails on E2B, this is why.

Like the pruned gpt-oss artifact in this account, the file doubles as a

compatibility probe: it fails loudly on runtimes that assume a scalar, and that

is a useful property rather than an inconvenience.

Measured

On an 8 GB Apple Silicon M1, Xyntetik Runner 0.1.13-alpha:

| | |

|---|---|

| Resident weights | 2.63 GB |

| Backend | Metal (zero-copy weights), 35 layers, ctx 4096 |

| Decode | 19.93 tok/s (greedy, --temp 0) |

| CPU vs Metal | byte-identical since runner v0.1.11, which implemented the E-series/PLE path on Metal |

2.63 GB is the number that matters on an 8 GB machine: it fits the Metal

working set with room for a KV cache, where a 5 GB E4B does not.

Measured fidelity vs BF16, added 2026-08-12

This exact file (sha 16ffaa2e...) was gated against its own BF16 parent:

400 teacher-forced positions, greedy, KLD over the union of both sides'

top-20 logprobs, zero-point self-check exact (0.0 KLD / 100% top-1 on the

parent vs itself).

| metric | this file vs BF16 | the project's publication bar |

|---|---|---|

| top-1 agreement | 77.75% | >= 97% |

| margin-qualified top-1 (bar v2, re-measured 2026-08-14) | 88.25% | >= 97% |

| mean KL divergence | 0.286 | <= 0.05 |

| mean top-8 overlap | 0.766 | none |

**This file does not meet the fidelity bar this account now applies to new

uploads.** It predates that bar. The Q4_K retention (see the corrected

section above) buys it roughly 20 top-1 points over a plain Q4_0 of the

same model, measured at 58.0% in the same session, and it still disagrees

with its parent on roughly 1 token in 4.5. What it remains good for:

smoke-testing a runtime on an 8 GB machine at 20 tok/s, where nothing

higher-precision fits at all. What it is not: a faithful gemma-4-E2B. If

you have the memory for the official Q4_K_M (3.46 GB), prefer it; fidelity

gates for the 3-5B Q4_K_M class are queued and will be published when

measured. Full context: the quantization-damage-vs-model-size finding in

the Xyntetik suite repo (q4_0-quality-gate-2026-08-12.md).

Running it

./runner -m gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf --serve --port 8080

Any GGUF runtime with gemma-4 E-series support that reads

feed_forward_length per layer will also load it.

Use the chat template. Instruction-tuned Gemma degenerates on raw

untemplated prompts, the official builds do this too, so judge quality

through /v1/chat/completions, not a bare completion prompt. Raw prompts are

fine for speed measurement and byte-identity comparisons, where both sides

degenerate equally and the comparison still holds.

Licensing

Apache 2.0. Gemma 4 ships under Apache 2.0, unlike Gemma 1-3, which carry

the custom Gemma Terms of Use with their Prohibited Use Policy. This card

previously said license: gemma, which was wrong: it named the older

generation's terms. Corrected 2026-08-09 after checking the licence for this

generation rather than the family.

If the Hub shows an imatrix tag on this repository, it is auto-derived and

inaccurate, no importance matrix was used. This is a plain requantization

pass (see the correction section for what "plain" turned out to mean).

Publication policy

This account publishes only artifacts that passed their quality gate;

experiments that failed ship as negative-result documentation in the

runner repo rather than as

uploads. Corrections to published cards are made in place and dated, this

card carries two (license, 2026-08-09; content labeling, 2026-08-11), because

the alternative is a record that quietly stops being true.

Measured envelope

Every artifact this account publishes ships a measured-envelope sidecar

(gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf.envelope.json, in this repo) — the same file

xyntetik-runner reads at

load: download it next to the GGUF and the runner reports the measured state at

load. A measured envelope is a **dated observation, not a standing quality

claim** — it records exactly what this file is, where it came from, and what

was measured, so the claims above stay traceable.

| | |

|---|---|

| artifact sha256 | 16ffaa2ea568e7a0c9b5c6ddbd39a376879f9ac007861334974de39f4cc52e3f |

| what changed | Q4_K_M → Q4_0 mix (only the parent's Q5/Q6/Q8 tensors moved; Q4_K retained) |

| reference (parent) | google/gemma-4-E2B-it BF16 (gated vs it; see table above) |

| fidelity vs parent | FAILS the bar — plain top-1 77.75%, margin-qualified 88.25%, mean KLD 0.286: a try-the-runner artifact, stated openly |

| tool calling | native gemma-4 protocol (truncation-safe) |

| measured | 2026-08-14 |

| verdict | experimental — loads with a banner; never a standing certification |