---

license: gemma

pipeline_tag: text-generation

base_model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced

tags:

• gguf
• long-context
• yarn
• gemma4
• uncensored
• mtp
• speculative-decoding
• vision
• llama.cpp
• ollama

---

<img src="banner.jpeg" width="720"/>

Gemma4-12B Uncensored: 1M Context + MTP + Vision

HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (12B dense, Google QAT checkpoint) with a 1,048,576-token context baked in (4x the native 262,144), shipping with its MTP speculative-decoding draft head and vision tower. All numbers below were measured on these exact files.

<table>

<tr>

<th style="background:#1a73e8;color:#fff;padding:8px 14px;">Capability</th>

<th style="background:#1a73e8;color:#fff;padding:8px 14px;">Status</th>

</tr>

<tr><td><b>1M context</b></td><td><b>Certified: 10/10 at every rung from 64K to 1M</b>, f16 KV, on a single RTX 5090</td></tr>

<tr><td><b>MTP speculative decoding</b></td><td>144.9 to 218.4 tok/s (<b>+51%</b>), acceptance 0.723 (measured on this trunk, RTX 5090)</td></tr>

<tr><td><b>Vision</b></td><td>Verified July 6, 2026: reads image text and identifies objects</td></tr>

<tr><td><b>Uncensored</b></td><td>HauhauCS Balanced abliteration; trunk weights bit-identical to the source release</td></tr>

</table>

Needle-in-a-haystack

<img src="niah_heatmap.png" width="640"/>

The first Gemma 4 we know of certified needle-perfect at 1,048,576 tokens. Thanks to Gemma's 5:1 sliding-window attention the KV cache stays small enough that the entire 1M certification ran on a 32 GB RTX 5090 at f16 KV. One cell (786K, depth 15 percent) missed on the first seed and passed 10/10 on a second seed; both runs are in results.jsonl.

MTP speculative decoding

<img src="mtp_speedup.png" width="480"/>

The draft head predicts ahead and the trunk verifies every token, so output is identical to standard decoding, only faster. Measured speedup on this uncensored trunk beats the ~35 percent claimed upstream.

Multi-hop retrieval: chains hold at 1,048,576 tokens

Beyond single-needle NIAH, we run a harder "hop" tier: chained lookups where each answer keys the next, at true 1M context. Thinking OFF, 5 chains per rung:

| Rung | Chains found |

|---|---|

| 131,072 | 5/5 |

| 524,288 | 5/5 |

| 1,048,576 | 5/5 |

Perfect chain retrieval at a million tokens. (The battery also probes a multi-step variable-tracking tier; our scorer for that tier is under review before we publish it, so those numbers are held back rather than reported unverified.) There is also a stretch rung at 1,572,864 tokens (1.5M, the extended-context sibling of this model) scoring 6/10 on single-needle NIAH: past the 1M bake it degrades, exactly as honestly expected.

RULER at long context (NVIDIA's benchmark, their scorer)

NIAH proves retrieval; RULER is the harder, industry-standard suite (multi-key retrieval, variable tracking, aggregation). We ran five RULER tasks at 131K on this exact GGUF, 25 samples per task, greedy-free vendor sampling (temp 1.0, top_p 0.95, top_k 64, seed 42), scored end to end by NVIDIA's own evaluate.py (RULER commit 38da79d). The bridge script and full reproduction recipe are published in aviary-1m/tools/ruler.

<img src="ruler_131k.png" width="720"/>

All columns thinking OFF (see the headline below for why). 25 samples per task, NVIDIA's own scorer.

| Task | 131K | 262K | 524K | ~1M* |

|---|---|---|---|---|

| niah_multivalue | 88.0 | 83.0 | 71.0 | 57.0 |

| niah_multiquery | 100.0 | 100.0 | 100.0 | 99.0 |

| variable tracking (vt) | 95.2 | 91.2 | 88.8 | 68.8 |

| common words extraction (cwe) | 80.0 | 46.8 | 13.6 | 5.2 |

| frequent words extraction (fwe) | 96.0 | 86.7 | 80.0 | 97.3 |

| Average | 91.8 | 81.5 | 70.7 | 65.5 |

\* The ~1M rung uses a nominal length of 917,504 with the full 1,048,576 context served. Reason, honestly: RULER sizes prompts with a reference tokenizer that undercounts this model's real tokenization by about 6 percent. Our first attempt at nominal 1,048,576 overflowed the context window and scored all-null on three tasks; that was a harness artifact, not a model failure, so we re-ran at a nominal length whose real token count (~976K) fits the served window. Zero nulls across all 125 samples in the run reported above.

The ladder climbs honestly to the top rung. Pure retrieval barely moves even at a million tokens: multiquery holds 99 and frequent-words extraction actually posts its second-best score of the whole ladder at 97.3. The cost of distance concentrates exactly where the rest of the ladder predicted: aggregation over the entire window (cwe, 5.2) and dense multi-value retrieval (57.0). Variable tracking finally bends at 68.8. This is what a real 1M model looks like: near-perfect needlework at full depth, honest degradation where the task requires holding the whole window at once. (For the record, thinking-ON at 131K averaged 45.9, half the thinking-OFF score.)

The honest headline: thinking mode halves this model's RULER score at 131K. The failure mode is specific and reproducible: on aggregation tasks (cwe/fwe) and some hard retrieval samples, the model reasons in circles and exhausts any generation budget (tested to 12K tokens) without ever emitting an answer. At temperature 0 the same samples loop deterministically; vendor sampling reduces but does not eliminate it. Those runs score 0 by RULER's rules and we report them as such, nulls and all.

Practical guidance: for long-context retrieval and aggregation work with this model, run thinking OFF. 91.8 average at 131K with zero unanswered samples is the model's real capability; thinking mode is the liability, not the context window. Raw prediction files, both conditions, are in the repo history for anyone who wants to re-score.

Files

| File | Size | Role |

|---|---|---|

| gemma4-12b-uncensored-1M-Q4.gguf | 7.4 GB | Trunk, 1M baked, QAT 4-bit |

| mtp-gemma-12b.gguf | 254 MB | MTP draft head, pair with -md |

| mmproj-gemma12b-hauhau.gguf | 175 MB | Vision tower, pair with --mmproj |

| niah_heatmap.png, mtp_speedup.png, results.jsonl | small | Verification evidence |

Every file, every mirror

Nothing was discontinued: every quant is one click away. Hugging Face carries the curated picks, ModelScope always carries everything, and Ollama serves ready-to-run tags.

On Ollama every tag ships with the vision tower bundled and the 1M rope metadata baked in.

| File | Size | Hugging Face | ModelScope | Ollama |

|---|---|---|---|---|

| gemma4-12b-uncensored-1M-Q4.gguf | 7.4 GB | download | download | ollama run satgeze/gemma4-12b-uncensored-1m |

| mmproj-gemma12b-hauhau.gguf | 175 MB | download | download | bundled in every tag |

| mtp-gemma-12b.gguf | 254 MB | download | download | - |

Run it

llama.cpp, everything on:

llama-server -m gemma4-12b-uncensored-1M-Q4.gguf \
  -c 1048576 -np 1 --jinja \
  -md mtp-gemma-12b.gguf --spec-type draft-mtp --spec-draft-n-max 3 \
  --mmproj mmproj-gemma12b-hauhau.gguf

Ollama (1M and vision work; Ollama has no speculative decoding yet, so the MTP head adds no speed there):

FROM ./gemma4-12b-uncensored-1M-Q4.gguf
RENDERER gemma4
PARSER gemma4
PARAMETER num_ctx 262144

The RENDERER and PARSER lines avoid imported-GGUF template bugs under tool-heavy use. Raise num_ctx as memory allows.

How to actually use a 1M-context model

Long context is a capability, not a magic mode. Habits that measurably help (from our own RULER and hop testing on this model):

1. Re-state your standing instructions near the end of long prompts. Recency beats depth.
2. Prefer one big reference dump over a long accumulated chat. Fresh session per task, context used as a library.
3. After any compaction or summarization, repeat your active rules yourself.
4. Run thinking OFF for retrieval and aggregation work. The RULER table above is unambiguous: thinking mode halves this model's score at long range.
5. Expect the extremes to cost time: prefill at 500K+ is slow on any hardware. Budget for it.

How this was built

YaRN rope-scaling metadata (factor 4.0 over native 262,144) baked into the GGUF header with gguf-py; weights are bit-identical to the HauhauCS release, no fine-tuning. Gemma 4's dual-rope design takes YaRN on its global-attention layers. Certification harness: 10 needles per rung at depths 5 to 95 percent, temperature 0, seeded prompts, f16 KV only. Method and tooling: github.com/satindergrewal/aviary-1m.

For base capability benchmarks see Google's official Gemma 4 cards; uncensoring quality versus the official trunk has not been independently benchmarked here.

Credits

Base model and QAT: Google (Gemma license; its terms flow down to these files). Uncensoring and packaging: HauhauCS. MTP head: Unsloth (via the HauhauCS repo). 1M YaRN extension, benchmarking, and certification: SatGeze.

Sister repos: 12B | 26B-A4B | 31B | Qwen3.6-35B

Mirrors: Hugging Face | ModelScope