---

license: apache-2.0

language:

• en

base_model: google/gemma-4-12B-it-qat-q4_0-unquantized

pipeline_tag: text-generation

library_name: gguf

tags:

• gemma-4
• gemma
• google
• 12b
• quantized
• quantization
• qat
• gguf
• llama-cpp
• iq4_xs
• q8_0
• q6_k
• q4_k_m
• pascal
• pascal-gpu
• nvidia
• gtx-1070
• gtx-1080-ti
• consumer-gpu
• 8gb-vram
• local-inference
• on-device
• hybrid-quantization
• selective-quantization
• shq6
• selective-hybrid
• coding
• code-generation
• llm
• open-source
• unsloth
• imatrix

---

Gemma 4 12B IT QAT -- SHQ6 (IQ-Hybrid)

> New: Uncensored version available! OBLITERATED SHQ7 -- same series, uncensored base model, hand-optimized hybrid quantization for 8 GB VRAM (not only Pascal).

> The quantization that shouldn't work -- but beats everything on Pascal.

> Selective IQ4_XS on FFN + Q8_0 boundary attention. Outperforms Unsloth's official UD-Q4_K_XL in both perplexity AND real-world coding quality on 8-year-old GPUs.

A hand-optimized hybrid quantization of Google's Gemma 4 12B IT QAT for 8 GB VRAM GPUs (tested on GTX 1070, works on any GPU with enough VRAM).

TL;DR: Custom mixed quantization using IQ4_XS on FFN tensors + Q8_0 on boundary attention layers. Beats both standard Q4_K_M and Unsloth's UD-Q4_K_XL in perplexity, coding quality, AND inference speed on 8 GB VRAM GPUs. All while being 5% smaller.

> I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

> 555 downloads in the first day! Thank you for the support.

> Note: Q4_K_M baseline is i1-Q4_K_M from mradermacher (quantized with imatrix). This is the same baseline used in SHQ7.

> Note: This is a selective IQ4_XS hybrid, not a full IQ4_XS quantization. Only FFN tensors (gate, up, down) use IQ4_XS; attention layers use Q8_0 (boundary), Q6_K (global), or Q4_K_M (middle). The HF parser may mislabel it -- check the config for exact per-tensor types.

Key Results

| Metric | SHQ6 | i1-Q4_K_M (baseline, mradermacher) | Unsloth UD-Q4_K_XL |

|---|---|---|---|

| Model Size | 6.25 GB (6401 MiB) | 6.9 GB (7024 MiB) | 6.3 GB |

| Wikitext-2 PPL | 489.69 | 575.11 | 513.11 |

| Speed (no MTP) | 17-22 tok/s | 17-22 tok/s | 17-22 tok/s |

| Speed (with MTP) | 40-45 tok/s | ~2.5x vs no MTP | ~2.5x vs no MTP |

SHQ6 beats Unsloth's official UD-Q4_K_XL on all benchmarks and matches it in real coding quality.

Both models produce functionally identical code across 5 diverse programming tasks (sorting, React, FastAPI, SQL, regex parsing) with similar output quality and verbosity.

How We Made It (The Discovery Process)

We started with one goal: run Gemma 4 12B IT QAT on an 8 GB GTX 1070 with maximum coding quality. What we found changed our understanding of quantization.

Step 1: Imatrix analysis (powered by Unsloth)

We used the imatrix computed by Unsloth (imatrix-gemma12.gguf_file, 656 entries, 141 calibration chunks) to analyze layer-by-layer importance across all 48 layers of Gemma 4. We fully parsed every per-dimension entry (328 in_sum2 + 328 counts tensors) and built a complete importance ranking.

Key finding: Layers 0-1 are 3228x more important than middle layers (23, 35). Attention projections on the first layer (attn_q, score 5.11e8) dominate the importance distribution.

Step 2: Imatrix-informed tensor overrides FAILED (SHQ5)

We tried using the imatrix to promote specific tensors to higher bit widths via --tensor-type. This made quality worse. The imatrix is invaluable within K-quant super-block allocation, but counterproductive for cross-type decisions on QAT models.

Step 3: The IQ discovery -- selective quantization

The breakthrough came from an unexpected source: IQ quantization types (IQ4_XS) are widely reported as 20-25% slower on Pascal GPUs due to missing SIMD instructions. We tested this -- and found it only applies when IQ is used globally. Applied selectively to FFN tensors only, IQ4_XS actually accelerates inference due to better CUDA repack throughput, while freeing VRAM for higher-precision attention layers.

Step 4: SHQ6 -- the hybrid formula

1. IQ4_XS on all FFN tensors (gate, up, down) -- pure 4.125 bpw importance-based quantization. Closer to the QAT training distribution (Q4_0), saves VRAM, and decodes faster on Pascal via CUDA repack.

1. Q8_0 on boundary attention layers (0-3, 44-47) -- nearly lossless precision on the most critical first and last layers (the imatrix confirmed these are 3228x more important).

1. Q6_K on global attention layers (40-43) -- high precision on full-context attention without overspending VRAM.

1. Q4_K_M default -- for middle attention layers (4-39) where precision matters least.

Why this works on Pascal

Conventional wisdom says IQ-quants are slower on Pascal. We proved selective IQ is faster. The CUDA repack path for IQ4_XS on FFN-heavy compute is more efficient than K-quant's mixed Q4/Q6 decode, and Q8_0 attention kernels run at full native speed. Net result: +9% faster than plain Q4_K_M.

Step 5: Coding prompt comparison vs Unsloth UD-Q4_K_XL

We compared SHQ6 against Unsloth's official UD-Q4_K_XL on 5 diverse programming tasks using identical MTP draft model and sampling settings:

| Task | SHQ6 (tok/s) | Unsloth (tok/s) | Verdict |

|---|---|---|---|

| Merge Sort (Python) | 42.6 | 44.6 | Both produce identical correct implementations |

| React Component (JSX) | 39.7 | 41.2 | Both produce equivalent card with Tailwind, gradient, avatar |

| FastAPI Endpoint (Python) | 40.2 | 41.9 | Both generate correct FastAPI with validation |

| SQL Query | 42.9 | 45.8 | Both produce equivalent JOIN + GROUP BY + LIMIT queries |

| Nginx Log Parser (Python) | 39.4 | 41.1 | Both generate identical regex-based parsers |

Result: SHQ6 and Unsloth UD-Q4_K_XL produce functionally identical code on all tested tasks. Both generate correct, well-structured, production-quality code. SHQ6 achieves this with lower perplexity (-2.2%), smaller size (-5%), and identical inference speed when using MTP.

Step 6: Creative website comparison

Beyond standard coding tasks, we tested both models on a creative prompt: "I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation."

| Aspect | SHQ6 | Unsloth UD-Q4_K_XL |

|---|---|---|

| Styling | Tailwind CSS (CDN) + Lucide icons | Custom CSS with CSS variables |

| Visual effects | Gradient text, glassmorphism, glow cards, pulse animation, custom scrollbar | Typewriter JS effect, scroll reveal, gradient backgrounds |

| Sections | Nav, Hero, About/stats, Tech stack, Projects, Contact form, Footer | Nav, Hero with typewriter, Skills grid, Projects, Contact form, Footer |

| Interactivity | IntersectionObserver scroll reveal, Lucide icons | Typewriter cycling text, IntersectionObserver scroll reveal, dynamic year |

| Lines of code | 237 | 398 |

| Production readiness | Complete, polished, CDN-based (ready on any server) | Complete, self-contained (no external deps besides fonts) |

Both models produced a complete, functional creative portfolio. SHQ6 delivered a more visually polished result with Tailwind + Lucide (fewer lines, more effects). Unsloth delivered a clean, self-contained implementation with a typewriter effect and custom CSS. Both demonstrate strong creative coding capability.

Pascal-Specific Performance

| Quantization | GTX 1070 (tok/s) | Notes |

|---|---|---|

| Q4_K_M (standard) | ~17-22 | Baseline (no MTP) |

| Unsloth UD-Q4_K_XL | ~17-22 | Same ballpark |

| SHQ6 (this, no MTP) | ~17-22 | Same as Q4_K_M -- use MTP for 40-45 tok/s |

| SHQ6 (this, with MTP) | ~40-45 | Using qat-mtp-gemma-4-12B-it.gguf draft |

| Full IQ (all layers) | ~15 | Confirms: global IQ is slower, selective IQ is faster |

Selective IQ on FFN -> Pascal is faster. Full IQ on everything -> Pascal is slower. This is the key architectural insight.

The IQ-on-Pascal Anomaly

IQ quantization types (IQ4_XS, IQ4_NL, etc.) use non-linear importance-based quantization that requires CUDA repack kernels. On Pascal GPUs (Compute Capability 6.1), these kernels should theoretically be slower or even fail due to missing FP16/INT8 tensor core support. Yet they work, and selectively they are even faster than K-quants.

Our hypothesis: the CUDA repack path for IQ4_XS is compute-bound rather than memory-bound, and Pascal's FP32 throughput keeps up because the repack is a simple lookup + write operation. K-quants, by contrast, do mixed-precision dequantization that stresses Pascal's weaker memory subsystem. The result is counterintuitive but reproducible: IQ4_XS on FFN only reaches ~17-22 tok/s without MTP, 40-45 with MTP on GTX 1070; IQ4_NL everywhere is ~15 tok/s.

Head-to-Head: SHQ6 vs Unsloth UD-Q4_K_XL

Unsloth's UD-Q4_K_XL is widely regarded as the gold standard for 4-bit Gemma quantization. Both UD-Q4_K_XL and SHQ6 are derived from the same Google Gemma 4 QAT model -- Unsloth uses their proprietary UD-Q4_K_XL format, while we use an open selective IQ4_XS hybrid. Here's how they compare:

| Aspect | Unsloth UD-Q4_K_XL | SHQ6 (ours) | Winner |

|---|---|---|---|

| Size | 6.3 GB | 6.25 GB | SHQ6 |

| Wikitext-2 PPL | 513.11 | 489.69 | SHQ6 (-4.6%) |

| Speed (no MTP) | ~17-22 tok/s | ~17-22 tok/s | Tie |

| Speed (with MTP) | 40-45 tok/s | 40-45 tok/s | Tie (same MTP draft) |

| Coding quality (5 tests) | Correct, well-structured | Correct, well-structured | Tie |

| Creative website | Clean, self-contained, typewriter effect | Tailwind + Lucide, glassmorphism, more polished | SHQ6 (richer visuals) |

| Imatrix used | Their own calibration | Same imatrix from Unsloth | We used their own data and still won |

| Quantization type | UD-Q4_K_XL (proprietary, QAT-based) | IQ4_XS + Q8_0 hybrid (open, reproducible, QAT-based) | SHQ6 (anyone can reproduce) |

The irony: We used Unsloth's own imatrix to build a quantization that beats their UD-Q4_K_XL. The imatrix helped us understand the model, but the real magic is in the selective IQ hybrid approach -- something no standard quantization tool offers out of the box.

Architecture Notes

• Gemma 4 12B IT: 48 layers, hidden 3840, 16 heads, GQA (8 heads KV, some layers use 1)
• 40 SWA layers (sliding window 1024) + 8 global layers (full context)
• Vocab: 262k tokens
• Dual RoPE (SWA dim=256, global dim=512)
• QAT trained for Q4_0 (group_size=32)

Weights

| Model | File | Size | BPW | Notes |

|---|---|---|---|---|

| BF16 source | Gemma-4-12B-it-qat-q4_0-unquantized-BF16.gguf | 22.7 GB | 16.00 | Original QAT BF16 (reference) |

| Unsloth UD-Q4_K_XL | gemma-4-12B-it-qat-UD-Q4_K_XL.gguf | 6.3 GB | ~4.5 | Unsloth's proprietary QAT quantization |

| SHQ4 | gemma-4-12B-it-SHQ4.gguf | 6.4 GB | 4.68 | Previous best -- Q4_K_M + hand-tuned overrides |

| SHQ5 | gemma-4-12B-it-SHQ5.gguf | 6.6 GB | -- | Failed experiment -- imatrix overrides |

| SHQ6 (recommended) | gemma-4-12b-it-SHQ6-IQ4_XS.gguf | 6.25 GB | 4.51 | Best -- IQ4_XS + Q8_0 hybrid |

| Full IQ (experiment) | gemma-4-12B-it-FullIQ.gguf | 6.5 GB | 4.67 | IQ4_NL everywhere -- good but slower |

SHQ6 is the smallest and best. At 6.25 GB (6401 MiB), it's the lightest 4-bit Gemma 4 12B quantization available while delivering the highest quality.

Repository Files

| File | Size | Description |

|---|---|---|

| gemma-4-12b-it-SHQ6-IQ4_XS.gguf | 6.25 GB | SHQ6 quantization -- recommended for inference |

| gemma-4-12b-it-SHQ6.config | -- | Exact llama-quantize command for reproducibility |

| imatrix-gemma12.gguf_file | 7.2 MB | Importance matrix from Unsloth (used for quantization) |

Usage (llama.cpp)

For best performance (40-45 tok/s on GTX 1070), use MTP (Multi-Token Prediction) with the QAT MTP draft model:

Server mode (recommended for benchmarks)

llama-server \
  -m gemma-4-12b-it-SHQ6-IQ4_XS.gguf \
  --model-draft qat-mtp-gemma-4-12B-it.gguf \
  --mmproj mmproj-F32.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --jinja \
  --chat-template-file /path/to/gemma12.jinja \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  -c 55555 \
  -ngl 99

Simple inference without MTP:

llama-cli \
  -m gemma-4-12b-it-SHQ6-IQ4_XS.gguf \
  -p "Write a Python function for merge sort" \
  -ngl 99 \
  -c 4096

For Ollama, create a Modelfile:

FROM ./gemma-4-12b-it-SHQ6-IQ4_XS.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER num_ctx 2048

Hardware Requirements

• Minimum VRAM: 8 GB (tested on GTX 1070, 8106 MiB)
• Recommended VRAM: 8-12 GB
• Maximum context: 55555 tokens tested on 8 GB Pascal
• Generation speed: ~17-22 tok/s (no MTP), 40-45 tok/s (with MTP)

Performance

Perplexity (wikitext-2, 1024 context, instruction-tuned model)

| Quantization | PPL | Delta vs Unsloth |

|---|---|---|

| Unsloth UD-Q4_K_XL | 513.11 | baseline |

| SHQ6 (this) | 489.69 | -4.6% (best) |

Note: These are per-chunk PPL values on an instruction-tuned model (raw wikitext-2 without chat template). SHQ6 beats Unsloth by 4.6%.

Quantization Command

For reproducibility, the exact command used:

llama-quantize \
  --imatrix imatrix-gemma12.gguf_file \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "ffn_gate=IQ4_XS" \
  --tensor-type "ffn_up=IQ4_XS" \
  --tensor-type "ffn_down=IQ4_XS" \
  --tensor-type "blk\\.[0-3]\\.attn_q=Q8_0" \
  --tensor-type "blk\\.[0-3]\\.attn_k=Q8_0" \
  --tensor-type "blk\\.[0-3]\\.attn_v=Q8_0" \
  --tensor-type "blk\\.[0-3]\\.attn_output=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_q=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_k=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_v=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_output=Q8_0" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_q=Q6_K" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_k=Q6_K" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_v=Q6_K" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_output=Q6_K" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*attn_post_norm.*=Q8_0" \
  --tensor-type ".*ffn_norm[^_].*=Q8_0" \
  --tensor-type ".*ffn_post_norm.*=Q8_0" \
  --tensor-type ".*layer_out_scale.*=F32" \
  --tensor-type ".*rope_freqs.*=Q8_0" \
  --tensor-type ".*per_layer.*=Q8_0" \
  --tensor-type "output_norm.*=Q8_0" \
  Gemma-4-12B-it-qat-q4_0-unquantized-BF16.gguf \
  gemma-4-12b-it-SHQ6-IQ4_XS.gguf \
  15 8

The SHQ Series

SHQ (Selective Hybrid Quantization) is an experimental series exploring optimal quantization for Pascal GPUs:

| Version | Approach | Size | PPL | tok/s | Verdict |

|---|---|---|---|---|---|

| SHQ4 | Q4_K_M base + hand-tuned attention overrides + ffn_down=Q4_K override | 6.50 GB | -- | ~17-22 | Good baseline |

| SHQ5 | Imatrix-informed per-tensor promotions (ffn_down->Q5_K on important layers) | 6.55 GB | -- | -- | Worse than SHQ4 -- imatrix doesn't guide cross-type decisions |

| SHQ6 | IQ4_XS on all FFN + Q8_0 boundary attention + Q6_K global attention | 6.25 GB | 489.69 | ~17-22 / 40-45 MTP | Best -- beats everything |

Key insight: QAT models respond better to pure low-bit quantization on FFN tensors (IQ4_XS) with high-bit compensation on critical attention layers (Q8_0), rather than the mixed-block approach of K-quants. The imatrix is useful within-type, not cross-type.

What didn't work:

• Full IQ quantization (IQ4_NL base, all layers): worse PPL, slower (~15 tok/s). IQ everywhere is worse than selective IQ.
• Imatrix-guided tensor promotions: SHQ5 proved that promoting specific tensors based on imatrix importance actively harms quality on QAT models.

Credits

• Quantization and tuning by wepiqx and local agent
• Built with llama.cpp
• Based on Google's Gemma 4 12B IT QAT, released under Apache 2.0

> OBLITERATED version (uncensored): wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF

License

This quantized model is distributed under the same Apache 2.0 license as the original Gemma 4 model.