---

license: apache-2.0

license_link: https://ai.google.dev/gemma/docs/gemma_4_license

base_model: google/gemma-4-31B-it-assistant

base_model_relation: quantized

quantized_by: Avifenesh

pipeline_tag: text-generation

library_name: gguf

tags:

• gguf
• mtp
• draft-model
• assistant
• speculative-decoding
• gemma4
• gemma-4
• memra
• nvfp4
• blackwell
• conversational

model-index:

• name: gemma-4-31B-it-assistant-MTP-GGUF

results:

- task:

type: text-generation

name: Speculative decoding for gemma-4-31B-it (assistant/MTP drafter, K=5, greedy)

dataset:

name: held-out agent-shaped prompt set (12 prompts, single stream, 5 interleaved repetitions per arm)

type: memra-heldout-agentic

metrics:

- name: draft acceptance rate (accepted/drafted), fixed greedy workload

type: acceptance-rate

value: 0.496

verified: false

- name: single-stream greedy decode tok/s with this drafter, RTX PRO 6000 Blackwell 96GB

type: throughput

value: 118.25

verified: false

- name: single-stream greedy decode tok/s, plain reference on the same cell

type: throughput

value: 58.17

verified: false

source:

name: memra serving A/B cell — one fresh boot per arm per repetition, speculative output gated byte-identical to plain greedy decode

url: https://github.com/avifenesh/memra

---

gemma-4-31B-it-assistant — MTP drafter GGUF (Q8_0 + NVFP4)

GGUF builds of google/gemma-4-31B-it-assistant,

the official 4-block assistant (MTP) drafter for

google/gemma-4-31B-it, quantized for

speculative decoding of gemma-4-31B-it GGUF trunks.

The assistant is a small draft head (4 layers, hidden 1024, tied embeddings, 262k vocab)

that computes only its own queries and reads the target model's KV cache

(attention_k_eq_v, all 4 layers KV-shared), so its per-draft-token cost is a small

fraction of a target decode step. The target model verifies every drafted token; the

emitted stream is the target's own output.

Files

| file | quant | bytes | sha256 |

|---|---|---:|---|

| gemma-4-31B-it-official-Q8_0-MTP.gguf | Q8_0 | 514,666,944 | f5a8758752b1195623a7e22f185b4445d61f19086eaaf9125b47c631ef7a9b66 |

| gemma-4-31B-it-official-NVFP4-MTP.gguf | NVFP4 (6.79 BPW; norms/embeddings kept high-precision) | 414,134,720 | f3374652c1d302cbcd2d72480042965233dea493b72e97d8c14a1dd223844d24 |

Both files carry the full 17,336-byte gemma-4 chat template in their GGUF metadata,

byte-identical to the source checkpoint's (verified on the exact published bytes —

quantization tooling can silently drop chat templates, so this was gated, not assumed).

How they were made

• The official google/gemma-4-31B-it-assistant bf16 checkpoint was converted to an

F16 GGUF with a byte-parity gate against the source weights (per-layer scalars

byte-equal; output norm exact on all 1024 rows).

• llama-quantize produced the Q8_0 and NVFP4 files from that F16 conversion

(no imatrix). NVFP4 is a Blackwell-native 4-bit float format; it was measured on

RTX PRO 6000 Blackwell (sm_120). On other hardware, use the Q8_0 file.

• The F16 conversion itself is deliberately not published as a drafter: a float

MTP head does not arm memra's gemma speculative route (float 2D matmul weights ride

a different kernel path); the quantized builds are the servable artifacts.

Measured results

All numbers are our own measurements of these exact files. Acceptance is

protocol-dependent — trunk quantization, sampling, workload shape and verification

policy all move it — so every number below states its protocol.

Acceptance / identity / throughput A/B (fixed workload)

Protocol: memra engine (v0.95.0/v0.96.0

candidates), greedy decoding, single stream, 1× RTX PRO 6000 Blackwell 96 GB;

trunk = a Q6_K-class GGUF build of gemma-4-31B-it; draft depth K=5; 12 held-out

agent-shaped prompts; interleaved ×5 with one fresh boot per arm per repetition;

correctness gate outranks speed — the speculative stream must reproduce plain greedy

decode byte-exactly.

| arm | acceptance (accepted/drafted) | tok/s median | byte-identity vs plain |

|---|---|---:|---|

| Q8_0 | 67/135 = 0.496, bit-reproducible every repetition | 118.0 | 30/30 |

| NVFP4 | 67/135 = 0.496, numerically identical to Q8_0 every repetition | 118.3 | 30/30 |

| plain reference (same cell, drafter attached, spec route off) | — | 58.2 | — |

The two quantizations are interchangeable on this protocol: acceptance is identical

to the count, outputs are byte-identical to plain decode in both arms, and throughput

is a wash (~2.03× plain on this cell). What NVFP4 buys is ~100.5 MB on disk/transfer

and ~96 MiB resident VRAM.

Hosted-endpoint A/B (production serving config, measured 2026-08-20)

Four fixed greedy probes (256-token budget) through our then-hosted gemma-4-31B-it

endpoint, minutes apart, Q8_0 vs NVFP4 under the identical serving configuration:

outputs byte-identical on 4/4 probes (content and reasoning channels); probe-set

acceptance 549/817 = 0.672 (Q8_0) vs 547/823 = 0.665 (NVFP4) — a wash.

Both files served that endpoint in production: the Q8_0 build until 2026-08-20,

then the NVFP4 build until the hosted gemma-4-31B-it endpoint was retired on

2026-08-21. The measurements above are dated production receipts, not claims

about a live service.

Lineage pairing matters

This is the official-lineage head (minted from the bf16

gemma-4-31B-it-assistant checkpoint). In our A/Bs it measured 0.58–0.60 prose

acceptance on a trunk built from the official gemma-4-31B-it weights, and 0.28–0.34

when cross-paired with a QAT-lineage trunk. Pair this drafter with trunks derived

from the official weights; QAT trunks want the QAT assistant head.

Run it with memra

memra serves gemma-4-31B-it with this drafter

— attach it with MEMRA_DRAFT and the gemma speculative route arms

automatically (K=5; engages on greedy, unconstrained, text-only sessions; speculative

output is gated byte-identical to plain decode):

MEMRA_MODELS="google/gemma-4-31b-it=/path/to/gemma-4-31B-it-<trunk>.gguf" \
MEMRA_DRAFT=/path/to/gemma-4-31B-it-official-NVFP4-MTP.gguf \
memra-server

Hosted inference

The hosted gemma-4-31B-it endpoint this drafter served was retired on

2026-08-21 (dated receipts above). The same lab runs a production inference

API at api.tiyuvta.ai — currently serving

Qwen3.8 27B at native 262,144-token context (OpenAI Chat Completions,

Responses, and Anthropic Messages on one endpoint, tool calling included),

on the same memra engine and exactness gates used for every number on this

card. Docs: inference.tiyuvta.ai/docs.

License

apache-2.0, inherited from

google/gemma-4-31B-it-assistant;

see the Gemma 4 license note

linked from the base model card.