---

base_model: ManniX-ITA/gemma-4-A4B-98e-v7-coder-it

tags:

- gguf

- imatrix

- quantized

license: apache-2.0

---

gemma-4-A4B-98e-v7-coder-it-GGUF

GGUF quantizations of ManniX-ITA/gemma-4-A4B-98e-v7-coder-it,

the loop-fixed code prune of Gemma 4 26B-A4B (128→98 experts/layer, ~20.8B).

All quants made using imatrix with calibration data v5.

The imatrix.dat used is included in this repo for reproducibility/audit; mmproj-gemma4.gguf

is the shared Gemma 4 SigLIP vision tower (untouched by pruning) for multimodal use.

Quantizations — score, size & bits-per-weight

Each published tier was scored on HumanEval+ (164) and MultiPL-E-100 (llama.cpp,

deployment sampler — min_p), as a quant-vs-quant comparison of how each tier holds the

model's code ability. The table shows, per tier, the code score, the exact size, and the **true

bits-per-weight (bpw = 8 × bytes ÷ 19,877,953,946). ⭐ marks a recommended pick**.

| Tier | Size | bpw | HE+ % | MPE-100 % |

|---|---:|---:|---:|---:|

| Q8_0 | 21.16 GB | 8.52 | 91.46 | 91.00 |

| Q6_K_L | 17.98 GB | 7.24 | 91.46 | 89.67 |

| Q6_K | 17.81 GB | 7.17 | 90.85 | 89.00 |

| Q5_K_L | 15.25 GB | 6.14 | 91.46 | 90.00 |

| Q5_K_M | 15.07 GB | 6.07 | 91.46 | 90.33 |

| Q4_K_L | 13.42 GB | 5.40 | 92.07 | 89.33 |

| Q4_K_M ⭐ | 13.24 GB | 5.33 | 92.68 | 89.00 |

| Q4_K_S | 12.21 GB | 4.91 | 92.68 | 89.33 |

| IQ4_NL | 11.42 GB | 4.60 | 89.63 | 88.67 |

| IQ4_XS | 11.01 GB | 4.43 | 91.46 | 88.33 |

| Q3_K_L | 10.94 GB | 4.40 | 90.85 | 89.00 |

| Q3_K_M | 10.51 GB | 4.23 | 91.46 | 87.00 |

| CD-Q2_K ⭐ | 8.82 GB | 3.55 | 89.63 | 86.00 |

Recommended picks:

• Q4_K_M ⭐ (13.24 GB) — recommended default — ties the best HE+ in the sweep (92.68%) at 13.2 GB.
• Q4_K_S ⭐ (12.21 GB) — same 92.68% HE+ at 12.2 GB if you want the smaller 4-bit K-quant.
• CD-Q2_K ⭐ (8.82 GB) — smallest tier still in the ~90% band — 89.63% HE+ / 86.0% MPE at 8.8 GB.

Reads: HE+ holds in the 89–93% band and MPE in the 86–91% band across the whole K-quant /

CD ladder — the 4-bit K-quants (Q4_K_M / Q4_K_S at 92.68%) actually top the sweep, and even the

3.55-bpw CD-Q2_K stays at ~90%. The CD-Q2_K ContribDynamic per-layer body is the

recommended low-bit path; the pruned MoE degenerates on a plain 2-bit or an IQ-family body at this

size (token-salad), so those tiers are not offered.

Head-to-head by file size — v7-coder vs Qwen2.5-Coder-14B (iso-disk)

Pairing by tier name is misleading — v7-coder is a ~20.8B-total MoE and Qwen2.5-Coder-14B is a

14.7B dense model, so the same tier name lands at a different file size. The fair comparison is

iso-disk: at a given GB budget, which model scores higher on HumanEval+? Qwen GGUFs are

bartowski's Qwen2.5-Coder-14B-Instruct-GGUF;

its ladder sits at 83–85% across the whole stack. v7-coder HE+ is the deployment-sampler

per-tier sweep above; Qwen HE+ is the same-stack reference.

| Disk band | Qwen2.5-Coder-14B (size / bpw / HE+) | v7-coder best (size / bpw / HE+) | Δ HE+ |

|---|---|---|---:|

| ~21.2 GB | (none — Qwen ceiling is Q8_0 15.70 GB) | Q8_0 21.16 / 8.52 / 91.46% | new top |

| ~17.8 GB | (none — Qwen ceiling is Q8_0 15.70 GB) | Q6_K 17.81 / 7.17 / 90.85% | new top |

| ~15.1 GB | Q8_0 15.70 / 8.54 / 84.76% | Q5_K_M 15.07 / 6.07 / 91.46% | +6.70 |

| ~13.2 GB | Q6_K 12.12 / 6.60 / 84.76% | Q4_K_M 13.24 / 5.33 / 92.68% | +7.92 |

| ~12.2 GB | Q6_K 12.12 / 6.60 / 84.76% | Q4_K_S 12.21 / 4.91 / 92.68% | +7.92 |

| ~11.0 GB | Q5_K_M 10.51 / 5.72 / 83.54% | IQ4_XS 11.01 / 4.43 / 91.46% | +7.92 |

| ~10.5 GB | Q5_K_M 10.51 / 5.72 / 83.54% | Q3_K_M 10.51 / 4.23 / 91.46% — iso-disk (same 10.5 GB) | +7.92 |

| ~8.8 GB | Q4_K_M 8.99 / 4.89 / 85.37% | CD-Q2_K 8.82 / 3.55 / 89.63% — ⭐ smallest ~90% | +4.26 |

Reads:

1. Iso-disk ~10.5 GB. v7-coder Q3_K_M (10.51 GB / 4.23 bpw / 91.46%) vs Qwen Q5_K_M (10.51 GB / 5.72 bpw / 83.54%): +7.92pp at the exact same file size, −1.49 bpw.
2. Sub-9 GB code-grade. CD-Q2_K (8.82 GB / 3.55 bpw / 89.63%) holds the ~90% HE+ band ~0.2 GB smaller and ~1.3 bpw lower than Qwen's best (85.37%, Q4_K_M, 8.99 GB).
3. Every band wins at lower bpw. Across the ladder the MoE uses 1.5–4 bpw less than the

dense Qwen tier at the same disk and still scores higher on HumanEval+ — the point of the iso-disk framing.

> CD recipes are open-source — generator at

> omnimergekit/scripts/generate_cd_maps.py.

How to Use

With llama.cpp:

llama-server -m gemma-4-A4B-98e-v7-coder-it-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-budget 8192 \
    --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.05 \
    --repeat-penalty 1.02 --repeat-last-n 2048

The --reasoning-budget flag is required for Gemma 4 thinking — without it the

model emits malformed channel tokens. Keep -c several times larger than the

reasoning budget — with -c equal to the budget the thinking phase can fill the

whole window and the answer degenerates. See Reasoning budget and thinking stop phrase below for the budget, the wrap-up phrase that stops reasoning

leaking into the answer, and why --reasoning-format plays no part in it.

Recommended sampling — --repeat-penalty 1.02 --repeat-last-n 2048 is the

field-tested sweet spot for long agentic / tool-calling sessions (validated with

opencode driving multi-turn coding work): it suppresses the intermittent

repetition loops without side effects. Stronger penalties (1.05–1.1) also stop

the loops but cause premature end-of-turn — the model announces a step and then

stops mid-task. Narrower windows (64–1024) at 1.02 let long-period loops through;

keep the full 2048. The remaining sampler values above are the Gemma 4 vendor

defaults baked into the GGUF, stated explicitly so CLI defaults don't override them.

With ollama: ollama pull mannix/gemma4-98e-v7-coder:Q4_K_M

(:latest = Q4_K_M; :vision-<tier> pairs the tier with the SigLIP vision tower).

Reasoning budget and thinking stop phrase (llama.cpp)

On a hard prompt this model will reason until it has consumed the whole

context window and then answer with nothing at all. llama.cpp can bound the

thinking block with a sampler, and — the part that actually matters — tell the

model why the block is being closed.

Needs llama.cpp b8508 or newer for the flags, b10091 or newer for the

per-request overrides.

Serve with a bounded thinking block

llama-server -m gemma-4-A4B-98e-v7-coder-it-Q4_K_M.gguf -c 32768 -ngl 99 \
    --jinja \
    --reasoning-budget 8192 \
    --reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n' \
    --temp 1.0 --top-k 64 --top-p 0.95 --min-p 0.05 \
    --repeat-penalty 1.02 --repeat-last-n 2048

| flag | meaning |

|---|---|

| --reasoning-budget N | -1 unrestricted (default), 0 close the block immediately, N > 0 cap it at N tokens |

| --reasoning-budget-message | text written into the block just before the closing tag is forced |

| --jinja | required — the delimiters come from the chat template (<|channel>thought … <channel|>). Without it llama.cpp has no tags to count and the budget silently does nothing |

Both flags also read from the environment: LLAMA_ARG_THINK_BUDGET and

LLAMA_ARG_THINK_BUDGET_MESSAGE.

--reasoning-format is not part of this. It only decides how the thinking

is handed back — message.reasoning_content versus left inline in

message.content — and never whether the budget is enforced: the delimiters the

sampler counts are set by the chat template regardless, so the cap binds under

auto, deepseek and none alike. The default auto already extracts

reasoning and is behaviourally identical to deepseek (they differ only in

name; the sole branch in the parser is != none). Leave it at the default so

the model's own tool-call and channel handling stays in play, and pin

deepseek only when a harness needs the thinking kept out of content.

--reasoning-budget on its own forces the closing tag the moment the budget

runs out, wherever the model happens to be. When that lands mid-thought the

model frequently does not register that it was interrupted: it carries on

reasoning, now inside the visible answer. The stop phrase is what prevents

that — it gives the model a reason to be finishing.

Two wordings that work

# "qwen" — the string Qwen's own service uses, from their docs
--reasoning-budget-message $'\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n'

# "voice" — shorter, in the model's own reasoning voice
--reasoning-budget-message $'\n\nOK, I have enough to answer now.\n'

Wording is model-specific: Qwen note that the ability to act on such a message

"is not explicitly trained but emerges naturally", so it is worth trying both

on your own workload. Leading and trailing newlines matter — they keep the

phrase off whatever half-finished line the cut landed on.

What it measures out to

Measured on the v7-coder IQ4_NL build of this family, served by the same

llama.cpp sampler. Three hard questions, temperature 0.6, fixed seed, answer characters with wall

time in brackets. Every run answered all three correctly, and thinking length is

unchanged by the message in every row:

| budget | no message | qwen | voice |

|---|---|---|---|

| 1024 | 2284 (35 s) | 1814 (26 s) | 1705 (26 s) |

| 2048 | 17411 (145 s) | 1557 (39 s) | 1673 (39 s) |

| 4096 | 1674 (68 s) | 1538 (67 s) | 1704 (68 s) |

AIME 2024, all 30 problems, budget 4096, -c 32768, vendor sampling:

| stop phrase | correct | answers over 20k chars | runs that hit the context wall | mean wall |

|---|---|---|---|---|

| none | 26/30 | 8 | 5 | 159 s |

| qwen | 22/30 | 1 | 0 | 76 s |

| voice | 25/30 | 1 | 0 | 82 s |

The phrase halves wall time and all but removes the runaway answers — single

rows go from 82,067 characters of answer to 1,655. The accuracy differences are

inside the noise at n = 30 (paired: qwen −4 net, voice −1 net, exact

binomial p ≈ 0.22 and ≈ 1.0), and the terse "Final Answer:" suffix from the s1

paper (arXiv:2501.19393) is not reproducing the accuracy collapse reported there

at this budget.

Per request, instead of per server

The server accepts both as request fields, overriding the command line:

{
  "messages": [ ... ],
  "thinking_budget_tokens": 8192,
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On the raw /completion endpoint the delimiters are not inferred, so they have

to be supplied with the budget:

{
  "prompt": "...",
  "reasoning_budget_tokens": 8192,
  "reasoning_budget_start_tag": "<|channel>",
  "reasoning_budget_end_tag": "<channel|>",
  "reasoning_budget_message": "\n\nOK, I have enough to answer now.\n"
}

On b10091 the message field must be present on /completion requests even

when empty: llama.cpp builds the sequence it forces from message + end_tag

inside that field's handler, so omitting it leaves the budget with nothing to

force — the sampler logs as though the cap fired while the thinking block stays

open.

Rules of thumb

• Keep -c several times larger than the budget. A budget equal to the context

lets the thinking phase fill the window on its own.

• A quarter of the context is a sensible starting point: 8192 at -c 32768.
• The budget is per thinking block, not per response — the sampler re-arms

when it sees a new opening tag, so a multi-turn agent gets a fresh window each

time.

Chat template & end-of-turn tokens (metadata refresh, 2026-07-30)

Every tier in this repo has been re-uploaded with corrected metadata — the rollout

completed 2026-07-30 and covers all 14 tiers: Q8_0, Q6_K_L, Q6_K, Q5_K_L,

Q5_K_M, Q4_K_L, Q4_K_M, Q4_K_S, IQ4_NL, IQ4_XS, Q3_K_L, Q3_K_M, CD-Q2_K,

F16. Every tier was verified after patching (template 19,177 B / md5 8119c2dc…,

eos=106, eot=1, bos=2, tool-call macros present, tensor count unchanged) and the

published file's sha256 re-checked against the patched file. The matching

mannix/gemma4-98e-v7-coder ollama tags were re-pushed as well — all 13 text tiers and

their 13 vision-<tier> counterparts (F16 is HF-only, by design).

The tensor payload is byte-identical — only the GGUF KV header changed — so sizes and

scores are unaffected, but you need to re-download to pick the fix up. A local file

still reporting eos_token_id = 1 (gguf_dump / llama-server startup log) is a stale

copy from before the refresh.

1. Chat template — agentic loop fix, rebased onto Google's current template. The

embedded tokenizer.chat_template is now our loop fix applied on top of Google's

current Gemma 4 template (upstream revision 2026-07-20, 18,683 B). The result is

19,177 B, md5 8119c2dcd5e62a4a6b79301ab13ac81d, and it is also published at the repo

root as chat_template.fixed.jinja for anyone serving

the bf16 weights.

The bug it fixes: the stock template **re-injects earlier assistant turns' thinking

content** back into the prompt on every turn. In long agentic / tool-calling sessions

that feeds the model its own reasoning back to itself and drives the repetition loops.

Google's current 18,683 B template is still affected — its thinking gate carries an

unconditional "index past the last user message" disjunct — so moving to a fresh

upstream template does not remove the need for this fix. The rebase leaves Google's

newer preserve_thinking flag intact (default false).

llama.cpp only uses the embedded template when you pass --jinja (added to the

command above). Without it llama.cpp falls back to its own built-in Gemma 4 formatter

and the fix does not apply. The recommended sampler above is still recommended — the

template fix and --repeat-penalty 1.02 address different halves of the loop problem.

2. End-of-turn tokens. eos_token_id is now 106 (<turn|>) and eot_token_id

1 (<eos>), which gives llama.cpp the end-of-generation set {106, 1, 50} —

matching generation_config.json ([1, 106, 50]) in the source repo. Previously every

tier shipped eos_token_id = 1 with no eot, so stopping depended on stop strings;

the model can now end its turn on its own token.

ollama caveat. ollama ≥ 0.32 formats Gemma 4 with a **compiled-in gemma4 renderer

and parser** and never reads the GGUF's jinja template, so the mannix/gemma4-98e-v7-coder

tags receive the EOG fix only. That is expected, not a defect: `ollama show

--modelfile printing TEMPLATE {{ .Prompt }} next to RENDERER gemma4 / PARSER

gemma4` is the correct state. For the template fix, serve the GGUF with llama.cpp

--jinja.

---

Gemma 4 A4B 98-Expert v7-coder — loop-fixed code prune (~20.8B)

> Eval complete (Q6_K / llama.cpp, greedy, same host). Every cell in the

> scoreboard is read from summary.json under the cohort-pinned greedy recipe

> (temperature 0.0, top_p 1.0, top_k 0). The 128e, v6-coder and v7-coderx columns

> are the matching same-host Q6_K runs. The GGUF and NVFP4A16 formats are deployment

> targets and are not separately benchmarked (cohort policy) — the Q6_K column

> is representative.

>

> Headline — the cohort's balanced code build. v7-coder leads the cohort on the

> easier-but-broad LiveCodeBench-medium slices (LCB-55-v4 98.18%, **LCB-100-v4

> 94.0%) and on HumanEval (98.17%), ties the cohort top on MATH-500 (95.0%)**,

> and takes AIME (80.0%). On the all-hard LiveCodeBench-77 set — the most

> discriminating LCB slice — it scores 84.42% (128e 79.22%, v7-coderx 85.71%),

> just behind the code-maximal sibling. This is the loop-fixed build: it

> force-keeps the agentic loop-protection experts and **replaces the earlier looping

> fs2440 prune**. Like its sibling it spends the prune budget on graduate science —

> GPQA-diamond sits at 51.52% (no targeted_gpqa term; ≈ v7-coderx 51.01%). If you

> need the hardest-code lean (all-hard LCB + HE+), see the sibling

> v7-coderx.

A research checkpoint that prunes the unpruned

Gemma 4 26B-A4B-it

(128 experts/layer, top-8 + shared, 30 layers) down to 98 experts per layer. The

fkbroad drop map (generate_drop_map_v5) up-weights generic-code (3×) and

LiveCodeBench-medium (2×) with no science or multilingual targeting, and

force-keeps the agentic loop-protection experts (agentic_eog, 46 experts, 0/46 dropped)

so the served model does not loop. Same 98e shape, same router, same attention, same norms as

the rest of the cohort, plus the mandatory shared-FFN α=1.2 upweight all coder

variants carry. No per-layer floor clamp and no DERN fold.

Benchmarks

Q6_K · llama.cpp · greedy (temperature 0.0, top_p 1.0, top_k 0), all four

models scored on the same host from summary.json. Row-max in bold.

This repo = v7-coder.

| Benchmark | 128e (unpruned) | v6-coder | v7-coder | v7-coderx |

|---|---:|---:|---:|---:|

| GPQA-diamond (198q) | 67.17 | 61.11 | 51.52 | 51.01 |

| AIME (30q) | 73.33 | 56.67 | 80.00 | 76.67 |

| MATH500 (100q) | 92.00 | 89.00 | 95.00 | 95.00 |

| GSM8K (100q) | 89.00 | 88.00 | 91.00 | 93.00 |

| ARC-Challenge (full) | 96.50 | 95.39 | 92.15 | 86.60 |

| IFEval (100q, strict) | 97.00 | 92.00 | 92.00 | 92.00 |

| HumanEval (164) | 97.56 | 98.17 | 98.17 | 96.95 |

| HumanEval+ (164) | 92.07 | 92.68 | 92.07 | 93.29 |

| LCB-medium-55 v4 | 96.36 | 92.73 | 98.18 | 92.73 |

| LCB-medium-100 v4 | 97.00 | 94.00 | 94.00 | 91.00 |

| MultiPL-E (100) | 90.00 | 89.00 | 89.67 | 89.00 |

<sub>Metrics: GPQA & GSM8K = exact_match flexible-extract · MATH500 = math_verify ·

ARC & AIME = exact_match · IFEval = prompt_level_strict_acc · HumanEval/+ = pass@1

chat-extract · LCB-55/100 & MultiPL-E = pass@1. 128e uses the lcb_medium_55/100

templates; the prunes use lcb_medium_*_v4 (corrected harness, equivalent task).</sub>

v7-coder is the balanced code sibling: it tops the cohort on LCB-medium and

HumanEval and ties on MATH/AIME, while v7-coderx leads the all-hard LCB-77 and HE+.

Both pay the budget on graduate science (GPQA ≈ 51) and the easier ARC axis.

LiveCodeBench across problem sets

The code score depends on the LiveCodeBench slice. All cells are the **same greedy

Q6_K / imat-Q6 llama.cpp stack** (build provenance verified per run); v4-55/100 mirror the

9-bench above. The all-hard 77q set is the most demanding and the most discriminating

across the cohort.

| LCB problem set | 128e | v7-coder | v7-coderx |

|---|---:|---:|---:|

| LCB-medium-55 (v4, 55q) | 96.36% | 98.18% | 92.73% |

| LCB-medium-100 (v4, 100q) | 97.00% | 94.00% | 91.00% |

| LCB-hard-77 (all-hard, 77q) | 79.22% | 84.42% | 85.71% |

Coder-field comparison — v7-coder vs Qwen2.5-Coder-14B / 7B + Qwen3.5-9B (Q6_K, llama.cpp, greedy)

The 9 canonical benches + MultiPL-E-100, all on the identical llama.cpp Q6_K / greedy

recipe (reasoning models served with `--reasoning-format deepseek --reasoning-budget 12288

--parallel 2`). Architectures differ — this is a same-harness comparison, not a same-class one:

• v7-coder — Gemma-4 26B-A4B MoE pruned to 98 experts (~20.8B total, ~A4B active), reasoning.
• Qwen2.5-Coder-14B / 7B-Instruct — dense, non-reasoning code specialists (bartowski Q6_K).
• Qwen3.5-9B — dense reasoning model (bartowski Q6_K).

| Bench (n) | v7-coder Q6_K | Qwen2.5-Coder-14B | Qwen2.5-Coder-7B | Qwen3.5-9B |

|---|---:|---:|---:|---:|

| ARC-Challenge-chat (1172) | 92.15% | 90.53% | 85.58% | 96.76% |

| GPQA Diamond flex (198) | 51.52% | 34.85% | 26.26% | 73.74% |

| GSM8K-100 flex | 91.00% | 89.00% | 80.00% | 79.00% |

| MATH-500-100 math_verify | 95.00% | 62.00% | 66.00% | 59.00% |

| AIME 2024 (30) | 80.00% | 10.00% | 10.00% | 56.67% |

| IFEval-100 (prompt_strict) | 92.00% | 68.00% | 54.00% | 93.00% |

| HumanEval-164 chat | 98.17% | 90.85% | 87.20% | 89.02% |

| HumanEval+-164 chat | 92.07% | 84.76% † | 83.54% | 80.49% |

| LCB-medium-55 v4 | 98.18% | 18.18% † | 12.73% | 58.18% |

| MultiPL-E-100 (macro) | 89.67% | 84.67% | 80.67% | 80.33% |

† Qwen2.5-Coder-14B HumanEval+ / LCB-medium-55 are the same-stack GGUF HE+ sweep numbers

(not re-run in this chain). All Qwen cells are the same-host reference runs used on the

v6-coder card — Qwen is a

fixed reference, so the columns are identical across the cohort; only the Gemma column changes.

> Note on Qwen3.5-9B. Qwen3.5-9B is a verbose, slow thinking model: it emits long

> <think> reasoning chains (often ≥1900 tokens even on a trivial GSM8K question), so it runs

> several× slower per question than the non-reasoning Qwen2.5-Coder models — well beyond what

> its 9B size would suggest. Its GSM8K / MATH-500 / GPQA cells were re-run after a harness fix

> (under batched, reasoning-parsed serving the verbose thinking intermittently left the final

> answer inside the reasoning block, mis-scored as empty content).

Recipe (summary)

98e prune from 128e via the fkbroad code recipe (generate_drop_map_v5: generic_code 3×,

targeted_lcb_medium_55 2×, all other targeting 0; target=98, protect_top=16, alpha=2.0,

strategy=max + breadth_bonus, no per-layer floor clamp), then the **agentic loop-protection

force-keep** (agentic_eog, 46 experts, 0/46 dropped — the loop fix that replaces fs2440), then

the mandatory shared-FFN α=1.2 upweight. No targeted_gpqa term and no DERN fold. v7-coder uses code/LCB weight

3×/2×; the sibling v7-coderx uses 4×/3×. Full recipe is on the

bf16 card.

Intended use & limitations

Compact (~13 GB at Q4_K_M, single 12–16 GB GPU) Gemma 4 checkpoint for **agentic

coding and code reasoning**. Serve with the reasoning parser enabled. A research prune,

not an official Google release; generic_multilingual is de-weighted (0×) and graduate

science (GPQA) is a budget axis (51.52% vs 128e 67.17%). Prefer Q4_K_M or higher.

Lineage

128e → (v4 → v5 → v6-coder code line) → v7 competence-map rebuild → **fkbroad code3/lcb2

selection + agentic loop-protection force-keep** = v7-coder (loop-fixed; supersedes fs2440).

Built and evaluated on the omnimergekit toolchain.