---

license: apache-2.0

base_model: google/gemma-4-31B-it

tags:

• gguf
• q4_0
• gemma4
• sublayer-removal
• structural-derivative
• xyntetik-runner

pipeline_tag: text-generation

---

Gemma 4 31B IT with attention block 48 removed (Q4_0, GGUF)

This file frees 64 MiB of KV cache at 4k context and 512 MiB at 32k, and

74.3 MB of file, by physically removing one attention sublayer, block 48 of

60, that a complete 180-row ablation map of this model measured as the

cheapest dispensable cut. It matches its parent to clean mean KLD 0.0223

with margin-qualified top-1 98.72% over 44,413 held-out positions

(measured 2026-08-30 on the zeroed-weights form of the same cut, carried to

this file by bit identity on 2026-09-04), and the runner's raw-protocol bar

on the Q4_0 pair reads KLD 0.0239, margin-qualified top-1 99.25%

(measured 2026-09-04). The KV figure is the point: it grows with context,

while the disk saving is 0.42% of the file.

<!-- BEGIN runner-toolcalling -->

Serve with xyntetik-runner &mdash; tool calls that still parse when the budget runs out

This GGUF is built to be served with xyntetik-runner, a single-binary, dependency-free C11 inference engine (CPU / CUDA / Metal, OpenAI-compatible server). Its one differentiator you cannot get from another engine:

> When a tool call runs past its token budget, Runner closes it to the smallest schema-legal document, so the arguments still parse.

Most OpenAI-compatible engines return finish_reason: "length" with an empty or malformed tool_calls the caller has to repair or retry. Runner does forced-truncation recovery: once a JSON document starts, it emits a legal ending when the budget expires. On local models &mdash; tight context, slow generation &mdash; that is the difference between an agent loop that finishes and one that retries from scratch.

Measured &mdash; what each engine hands the caller when the token budget cuts a tool call short (same box, same tool schema, same prompt, tool_choice:"required", temperature 0, budgets 1&rarr;64):

| engine | budget too small (1&ndash;16 tokens) | enough budget (64, control) |

|---|---|---|

| Runner | executable tool_calls, arguments parse | completes |

| vLLM 0.27.1 | no call; protocol framing leaks into content | completes |

| llama.cpp b10488 | no call; leak, then tool_calls with unparseable args | completes |

| Ollama 0.32.14 | no call; empty content, then HTTP 500 | completes |

| TensorRT-LLM 1.2.1 &dagger; | no call; <tool_call> leak, then empty content | completes |

| SGLang 0.5.17 &dagger; | no call; <tool_call> leak, then empty content | completes |

Every engine completes at 64 tokens &mdash; the failure is truncation, not misconfiguration. Below that, only Runner returns an executable call. This is the behaviour across every OpenAI-compatible engine measured, not a claim about engines not measured. &dagger; TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute; truncation recovery is a property of the runtime, not the model.

Constrained decoding also holds the shape of a tool call under quantization: on a full quant ladder, schema conformance and tool selection stayed 100% down to Q4_0 (argument agreement decayed to 50%) &mdash; it guarantees the shape of a call at any quantization, not its contents.

Quickstart

# build the single binary (no deps), then serve this GGUF:
runner -m gemma-4-31B-it-attn48-removed-q4_0.gguf --gpu off --serve
# point any OpenAI-compatible client at http://localhost:8080/v1

Full recipe, raw responses, and the per-release regression gate: the truncation benchmark.

<!-- END runner-toolcalling -->

> Read this first. This is a research artifact for people measuring

> sublayer removal, not a smaller or faster daily driver. It loads only in

> xyntetik-runner

> v0.4.7 or later (released 2026-09-04). Runner v0.4.6 and earlier, and

> stock llama.cpp (measured on b10076), refuse it with

> missing tensor 'blk.48.attn_q.weight'. That is

> the intended failure: the absence is declared in the header, never hidden.

> Today only the runner's CPU path omits the removed block, so pass

> --gpu off; the device paths are open work.

Run it

The file is 17.6 GB and runs on the CPU path, so it wants a machine with

roughly 20 GB of free memory and many cores. On a laptop this is a

measurement subject, not a chat model.

# a release binary (Linux x86-64 shown; macOS arm64 and Windows are on the same page)
curl -L https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/download/v0.4.7/runner-linux-x86_64.tar.gz | tar xz
curl -L -o gemma-4-31B-it-attn48-removed-q4_0.gguf \
  https://huggingface.co/Joakimpalm-Zen/gemma-4-31B-it-attn48-removed-Q4_0-GGUF/resolve/main/gemma-4-31B-it-attn48-removed-q4_0.gguf
./runner-linux-x86_64 -m gemma-4-31B-it-attn48-removed-q4_0.gguf --gpu off -v -p "hi" -n 32

The -v banner prints sublayers removed attn:48 and a kv cache line

67.1 MB smaller than the parent's at the default 4,096 context.

What was removed, exactly

| tensor | type | shape | bytes |

|---|---|---|---|

| blk.48.attn_q.weight | Q4_0 | [5376, 8192] | 24,772,608 |

| blk.48.attn_output.weight | Q4_0 | [8192, 5376] | 24,772,608 |

| blk.48.attn_k.weight | Q4_0 | [5376, 4096] | 12,386,304 |

| blk.48.attn_v.weight | Q4_0 | [5376, 4096] | 12,386,304 |

| blk.48.attn_q_norm.weight | F32 | [256] | 1,024 |

| blk.48.attn_k_norm.weight | F32 | [256] | 1,024 |

| total tensor payload | | | 74,319,872 |

Block 48 keeps its pre-attention norm, post-attention norm and its whole

FFN. The absence is declared the way llama.cpp's own per-layer arrays can

declare it: gemma4.attention.head_count and head_count_kv are per-block

arrays with a 0 at block 48 (32 and 16 elsewhere), the reading llama.cpp's

Nemotron-51B "deci" graph uses for attention-free blocks. The other 827

tensors are byte-identical to the parent.

| | parent | this file |

|---|---|---|

| file | 17,651,001,568 B | 17,576,681,600 B |

| tensors | 833 | 827 |

| KV cache at ctx 4,096 (f16) | 3,691.0 MB | 3,623.9 MB |

| KV cache at ctx 32,768 (f16) | 29,527.9 MB | 28,991.0 MB |

• File: gemma-4-31B-it-attn48-removed-q4_0.gguf
• SHA256: d7eaef9a8c05d8ad5cc63e4b37767113498b6ec6a31863519f5399f12cdb3134
• Parent: google/gemma-4-31B-it-qat-q4_0-gguf, file gemma-4-31B_q4_0-it.gguf,

revision 59dde245, SHA256 179cfb99212709597eae5929112cfca677e1bbf566178b479ae1da0c4772874b

(Google's own QAT Q4_0 release of google/gemma-4-31B-it)

Why block 48

A complete ablation map of the 31B decoder (60 blocks, whole-block, MLP-only

and attention-only cuts, 180 rows) was scored against the frozen BF16 parent

on a held-out split with the house bar of clean mean KLD at or under 0.05 and

margin-qualified top-1 at or above 97%. Five attention-only cuts passed, all

in the late band; attn:48 was the cheapest at KLD 0.0223. Cuts do not

combine (every pair fails the bar), so the deployable figure is one cut, and

this is it. The masked, zeroed-weights and removed forms of the cut are the

same function: the zeroed and removed files score bit-identically over 4,562

positions (maximum absolute logprob difference 0.0), and the zeroed file

scored bit-identically to the map's mask row at float64.

Measured envelope

| field | value |

|---|---|

| sha256 | d7eaef9a8c05d8ad5cc63e4b37767113498b6ec6a31863519f5399f12cdb3134 |

| derivation | the Q4_0 parent with the block-48 attention sublayer removed by xyntetik-runner --remove-sublayer attn:48 |

| reference sha256 | 179cfb99212709597eae5929112cfca677e1bbf566178b479ae1da0c4772874b |

| fidelity vs the BF16 parent (by identity with the zeroed form, 2026-08-30) | clean mean KLD 0.022272, margin-qualified top-1 98.7188%, top-1 agreement 96.4808%, 44,413 held-out positions. House bar KLD at or under 0.05 and margin-q at or above 97%: PASS |

| fidelity vs the Q4_0 parent (this file, 2026-09-04) | raw-protocol mean KLD 0.02394, margin-qualified top-1 99.25%, plain top-1 92.75%, top-8 overlap 0.928, 400 positions, tie band 0.5 nat: PASS |

| absolute check (by identity) | next-token top-1 66.3115% against the parent's 66.3292% over 45,045 positions, a difference of 0.018 points |

| KV cache freed (f16) | 67,108,864 B at ctx 4,096; 536,870,912 B at ctx 32,768 |

| file | 17,576,681,600 B, 74,319,968 B less than the parent |

| tool calling | native gemma4 protocol through the runner's OpenAI-compatible endpoint, correct call emitted (measured on the zeroed form) |

| runtime | xyntetik-runner main ad145bf, CPU path, 32 threads, 128-core x86 host |

| measured | 2026-09-04 (this file), 2026-08-30 (the zeroed form) |

| verdict | experimental |

Limits, read before quoting

• Parent-agreement is not a capability benchmark. Every number above

says how closely this file tracks its own parent on a held-out corpus, not

how well either model performs on a task. This project has published a

case where a model cleared the same bar and still failed a paired

benchmark test.

• "No quality loss" is not claimed and is not supported by this evidence.

The artifact disagrees with its parent on about 3.5% of held-out positions.

• Runner-only, CPU-only, for now. Any other runtime refuses the file by

name. The runner's GPU paths refuse it too until they learn to skip a

removed block.

• The disk saving is the small end of this program's results (0.42% of

the file). The reason to ship the file is the KV cache, which is 90% of

the disk saving at 4k context and seven times it at 32k.

• Two fidelity rows, two protocols. The BF16 house number comes from the

research scorer on a decontaminated held-out split; the Q4_0 row comes

from the runner's raw-completions script on its mixed fixture corpus at

400 positions. They are quoted beside each other, not as each other.

Reproduce

# the parent, from Google's repo
curl -L -o gemma-4-31B_q4_0-it.gguf \
  https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf/resolve/main/gemma-4-31B_q4_0-it.gguf
sha256sum gemma-4-31B_q4_0-it.gguf   # 179cfb99...

# the cut; survivors are copied byte for byte, so the output hash reproduces
./runner -m gemma-4-31B_q4_0-it.gguf --quantize gemma-4-31B-it-attn48-removed-q4_0.gguf \
  --remove-sublayer attn:48
sha256sum gemma-4-31B-it-attn48-removed-q4_0.gguf   # d7eaef9a...

# the Q4_0-pair fidelity row
python3 scripts/kld-compare-raw.py --model-a gemma-4-31B_q4_0-it.gguf \
  --model-b gemma-4-31B-it-attn48-removed-q4_0.gguf --runner ./runner \
  --corpus tests/fixtures/mixed-corpus.txt --max-positions 400 --out kld.json

Provenance

SURGERY.json beside the file records the parent revision and hash, the

exact tensors removed, the byte counts, the header declaration, the recipe

and the measurements that justified the cut. The .envelope.json sidecar is

the runner-readable measured envelope. The removal mechanism, its gates and

the measurements on this file are documented in the runner repository under

docs/sublayer-removal.md.