---

base_model: ambrosfitz/gemma4-ushistory-sft-parityfix

tags:

• gguf
• gemma

---

gemma4-ushistory-sft-parityfix-GGUF

GGUF quantizations of ambrosfitz/gemma4-ushistory-sft-parityfix

(the merged 16-bit parity-fix checkpoint), for local inference via llama.cpp / LM

Studio. Two quants are provided:

| File | Size | Notes |

|---|---|---|

| gemma4-ushistory-parityfix-Q4_K_M.gguf | 7.38 GB | Verified clean 8/8 on structured-output smoke test |

| gemma4-ushistory-parityfix-Q5_K_M.gguf | 8.55 GB | Verified clean 8/8 on structured-output smoke test |

Both fit comfortably on a 12GB+ GPU with full offload and room for a 4096-token KV

cache.

Required generation recipe

This model was trained and evaluated with a specific chat template and stop-token

setup. Getting either wrong produces malformed or garbage output — this was

confirmed directly during quantization verification (see below).

• Chat template: gemma-4, non-thinking mode. The correct jinja template is

embedded in this GGUF's own metadata (tokenizer.chat_template) — **do not

override it** with a runtime's built-in "gemma" preset. llama.cpp ships a

built-in template named gemma that is for the classic Gemma 1/2/3 format

(<start_of_turn>/<end_of_turn>) and is incompatible with this checkpoint's

Gemma-4-unified format. Passing --chat-template gemma to llama-server

silently overrides the correct embedded template and produces completely

unrelated, garbage output (confirmed: 0/8 parseable outputs, all hit the token

budget without stopping). Leave the chat template on its default (LM Studio:

don't override it; llama.cpp: don't pass --chat-template) so the runtime uses

the template embedded in this file.

• Stop token: <turn|> (vocab id 106). **This is not encoded as an EOS/EOG

token in the GGUF metadata** — the only marked EOS token is <eos> (id 1), and

llama.cpp's own automatic EOG detection at load time does not pick up <turn|>

either. This mirrors the original HF Transformers inference recipe, which also

had to pass it explicitly (`eos_token_id=[tokenizer.eos_token_id,

id_of("<turn|>")]). **You must set <turn|>` as an explicit stop string in your

runtime** (LM Studio: add it under the model's stop-string/sequence setting) —

without this, generation will run past the intended turn boundary and repeat

into garbage on longer outputs.

Verification

Both quants were served on-pod via llama-server and smoke-tested with 8 items

spanning all 6 Bloom's-taxonomy levels (remember/understand/apply/analyze as

multiple-choice, analyze/evaluate/create as open-ended — matching this model's

trained item-type schema), using the project's exact prompt construction and an

explicit <turn|> stop string.

**Result: 8/8 parsed, 8/8 structurally valid, 8/8 stopped cleanly — on both

quants.** No malformed or garbage output on either quant once the chat-template

issue above was fixed. Full per-item results and methodology in this repo's

source project (handoff_A_verification.md).

A small parity sanity check (word-count spread across MCQ options, this project's

max-min length-parity metric) on the n=5 MCQ smoke items per quant showed the

same pattern already documented for the un-quantized model: understand-level

items carry more residual parity failures than remember/apply. At this sample

size this is noise-level, and there is no evidence quantization itself degrades

parity beyond what the fp16 checkpoint already exhibits.

Recommended local config (tested target: RTX 4070, 12GB)

• Full GPU offload (-ngl 99 or LM Studio's "GPU offload: max")
• Context length: 4096 (matches training: passage + up to 1024 generated tokens)
• Chat template: leave on default / embedded (see above)
• Stop string: add <turn|> explicitly (see above)
• temperature: 0 for deterministic output, matching how this model was trained

and evaluated throughout this project

Lineage

• Base: Gemma 4 (unsloth/gemma-4-12b)
• SFT: ambrosfitz/gemma4-ushistory-sft → parity-fix continued SFT →

ambrosfitz/gemma4-ushistory-sft-parityfix (merged 16-bit) → this GGUF

quantization