---

license: apache-2.0

language:

• en

base_model: OBLITERATUS/Gemma-4-12B-OBLITERATED

pipeline_tag: text-generation

library_name: gguf

tags:

• gemma-4
• gemma
• google
• 12b
• quantized
• quantization
• gguf
• llama-cpp
• iq4_xs
• q8_0
• q6_k
• pascal
• pascal-gpu
• nvidia
• gtx-1070
• gtx-1080-ti
• consumer-gpu
• 8gb-vram
• local-inference
• on-device
• hybrid-quantization
• selective-quantization
• shq7
• lightweight
• uncensored
• obliterated
• coding
• code-generation
• llm
• open-source
• imatrix

---

Gemma 4 12B OBLITERATED -- SHQ7 (IQ-Hybrid)

> OBLITERATED + selective IQ hybrid for 8 GB Pascal GPUs. Uncensored coding model. 7.7% smaller than Q4_K_M at the same quality.

>

> WARNING: This is an uncensored model. It may generate content that other models refuse. Use responsibly. The maintainer assumes no responsibility for misuse.

>

> Note: The model file is named SHQ7-IQ4_XS.gguf for HF parser compatibility. This is NOT a pure IQ4_XS quantization -- it's a hybrid using IQ4_NL, Q4_K, IQ4_XS, and Q8_0 across different layers. Check the config for exact per-tensor types.

>

> Primarily optimized for coding tasks. May not perform well on general knowledge or non-code questions.

A hand-optimized hybrid quantization of OBLITERATUS/Gemma-4-12B-OBLITERATED for 8 GB VRAM GPUs (tested on GTX 1070, works on any GPU with enough VRAM).

> I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

TL;DR: Custom mixed quantization using IQ4_NL on boundary FFN + Q4_K on critical middle FFN + IQ4_XS on all other FFN + Q8_0 on all attention. Fits in 8 GB VRAM with room for KV cache.

Key Results

| Metric | SHQ7 | i1-Q4_K_M (baseline) |

|---|---|---|

| Model Size | 6.3 GB (6480 MiB) | 6.9 GB (7024 MiB) |

| Wikitext-2 PPL | 575.89 | 575.11 |

| Quantization Type | IQ4_NL + Q4_K + IQ4_XS + Q8_0 | Q4_K_M |

> Q4_K_M baseline quantized with mradermacher's imatrix (i1 importance matrix).

SHQ7 achieves Q4_K_M quality at 7.7% smaller size (544 MiB saved).

How We Made It

The Problem

Our goal was to create a quantization of OBLITERATED that matches Q4_K_M quality while being smaller. We tried 6 different approaches, failed 4 times, and found 2 winners.

Step 1: Q4_K_M baseline

Standard Q4_K_M: 7024 MiB, PPL 575.11. Fits in 8 GB but leaves little room for KV cache at large context.

Step 2: SHQ6 approach from QAT -- FAILED on OBLITERATED

We tried the SHQ6 formula (IQ4_XS on all FFN + Q8_0 boundary attention) that worked great on QAT models. On OBLITERATED it produced +8.6% PPL degradation (624.73). The OBLITERATED model responds differently to quantization than QAT models.

Step 3: IQ-gradient -- CATASTROPHIC FAILURE

We tried a gradient approach: IQ4_NL on boundary layers, IQ4_XS on middle, IQ3_XXS on least important layers. The IQ3_XXS (3.06 bpw) on 36 middle layers destroyed quality: +43.5% PPL (825.04). IQ3_XXS is catastrophic on any non-QAT model.

Step 4: Imatrix analysis

We used mradermacher's imatrix (656 tensors with per-dimension importance) to understand which layers actually matter:

| Layer | FFN Importance | Verdict |

|---|---|---|

| L0 | 2.48e+11 (#1) | CRITICAL -- every tensor in L0 is the most important in the entire model |

| L1 | 1.34e+11 (#5) | CRITICAL |

| L10 | 1.26e+11 (#6) | CRITICAL |

| L37 | 1.09e+11 (#7) | CRITICAL |

| L5, L36 | 4-6e+10 | Important |

| L25-31 | 8-12e+09 | Low importance |

Key finding: L0 is 87% of total importance. L0 FFN down (2.48e+11) is the single most important tensor in the entire 23 GB model. L1 FFN down (1.34e+11) is #5. Layer 0 attention (attn_q/k/v) are each 2.41e+11.

Step 5: IQ4XS_lean -- first success

We combined three strategies:

• Q8_0 on L0-1 FFN+Attn (protect the most critical layers)
• Q4_K_M on 11 important middle FFN (L5,10,25-31,36,37)
• IQ4_XS on all other FFN (efficient compression)
• Q8_0 on L40-47 attention (boundary/global)

Result: 6793 MiB, PPL 578.63 (+0.6% vs Q4_K_M). Close but not better.

Step 6: IQ4XS_lean_v2 -- TIED with Q4_K_M

We reduced the protected middle layers from 11 to 4 (L5,10,36,37) and kept IQ4_XS on L25-31:

Result: 6793 MiB, PPL 575.05 -- statistically tied with Q4_K_M (575.11) but 231 MiB smaller.

Step 7: IQ4XS_lean_v3 -- THE WINNER

We combined all optimizations:

• IQ4_NL on L0-1 FFN (non-linear quantization for #1/#2 most important tensors)
• Q4_K on L5,10,36,37 FFN (K-quant on 4 critical middle layers)
• IQ4_XS on all other FFN (efficient compression)
• Q8_0 on ALL boundary/global attention (L0-3, L40-47)

Result: 6480 MiB, PPL 575.89 -- same quality as Q4_K_M at 544 MiB smaller.

Step 8: Q6_K upgrade -- WORSE

We tried upgrading L0-1 FFN from IQ4_NL to Q6_K (6.56 bpw) and L5,10,36,37 FFN from Q4_K to Q5_K. Despite higher bit width, PPL went UP to 590.64 (+2.2%). IQ formats work better than K-quants when used with imatrix.

What We Learned

| Lesson | Details |

|---|---|

| IQ4_XS on ALL FFN fails on non-QAT | +8.6% PPL. QAT models respond differently. |

| IQ3_XXS is catastrophic anywhere | +43.5% PPL. Never use below IQ4_XS. |

| L0 is the most important layer | 87% of total importance. Every tensor matters. |

| IQ beats K-quant with imatrix | Q6_K on L0-1 was WORSE than IQ4_NL despite higher bpw. |

| Protect only truly critical middle layers | L5,10,36,37 need Q4_K. L25-31 are safe with IQ4_XS. |

| Boundary attention needs Q8_0 | L0-3 and L40-47 attention must be high precision. |

About OBLITERATED

This model is based on OBLITERATUS/Gemma-4-12B-OBLITERATED, an uncensored variant of Gemma 4 12B IT.

What "uncensored" means:

• The model will generate content that standard models refuse
• It includes safety disclaimers after generating restricted content
• It is primarily optimized for coding tasks
• It may not perform well on general knowledge questions

Architecture Notes

• Gemma 4 12B IT: 48 layers, hidden 3840, 16 heads, GQA (8 heads KV, some layers use 1)
• 40 SWA layers (sliding window 1024) + 8 global layers (full context)
• Vocab: 262k tokens
• Dual RoPE (SWA dim=256, global dim=512)
• Based on OBLITERATED (uncensored) model

Weights

| Model | File | Size | BPW | Notes |

|---|---|---|---|---|

| BF16 source | Gemma-4-12B-OBLITERATED-BF16.gguf | 23 GB | 16.00 | Original OBLITERATED BF16 |

| i1-Q4_K_M (baseline) | gemma-4-12B-OBLITERATED-Q4_K_M.gguf | 6.9 GB | 4.95 | i1 quantization with mradermacher's imatrix |

| SHQ7 (recommended) | gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf | 6.3 GB | 4.57 | Best -- IQ4_NL + Q4_K + IQ4_XS + Q8_0 |

Repository Files

| File | Size | Description |

|---|---|---|

| gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf | 6.3 GB | SHQ7 quantization -- recommended for inference |

| gemma-4-12B-OBLITERATED-SHQ7.config | -- | Exact llama-quantize command for reproducibility |

Quantization Command

For reproducibility, the exact command used:

llama-quantize \
  --imatrix imatrix-gemma4-obliterated.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "blk\.[01]\.ffn_gate=IQ4_NL" \
  --tensor-type "blk\.[01]\.ffn_up=IQ4_NL" \
  --tensor-type "blk\.[01]\.ffn_down=IQ4_NL" \
  --tensor-type "blk\.[01]\.attn_q=Q8_0" \
  --tensor-type "blk\.[01]\.attn_k=Q8_0" \
  --tensor-type "blk\.[01]\.attn_v=Q8_0" \
  --tensor-type "blk\.[01]\.attn_output=Q8_0" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_gate=Q4_K" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_up=Q4_K" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_down=Q4_K" \
  --tensor-type "ffn_gate=IQ4_XS" \
  --tensor-type "ffn_up=IQ4_XS" \
  --tensor-type "ffn_down=IQ4_XS" \
  --tensor-type "blk\.[0-3]\.attn_q=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_k=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_v=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_output=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_q=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_k=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_v=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_output=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_q=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_k=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_v=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_output=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*attn_post_norm.*=Q8_0" \
  --tensor-type ".*ffn_norm[^_].*=Q8_0" \
  --tensor-type ".*ffn_post_norm.*=Q8_0" \
  --tensor-type ".*layer_out_scale.*=F32" \
  --tensor-type ".*rope_freqs.*=Q8_0" \
  --tensor-type ".*per_layer.*=Q8_0" \
  --tensor-type "output_norm.*=Q8_0" \
  Gemma-4-12B-OBLITERATED-BF16.gguf \
  gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf \
  15 8

All Experiments (OBLITERATED)

| Experiment | Config | Size | PPL | vs Q4_K_M | Verdict |

|---|---|---|---|---|---|

| i1-Q4_K_M (baseline) | i1 with mradermacher's imatrix | 7024 MiB | 575.11 | -- | Baseline |

| SHQ6 on OBLITERATED | IQ4_XS all FFN + Q8_0 boundary | 6401 MiB | 624.73 | +8.6% | FAIL |

| IQ-gradient | IQ4_NL + IQ4_XS + IQ3_XXS | 6335 MiB | 825.04 | +43.5% | CATASTROPHIC |

| IQ4XS_lean | Q8_0 L0-1 + Q4_K_M middle + IQ4_XS rest | 6793 MiB | 578.63 | +0.6% | Close |

| IQ4XS_lean_v2 | Q4_K on L5,10,36,37 + IQ4_XS L25-31 | 6793 MiB | 575.05 | same | WIN |

| IQ4XS_lean_v3 | IQ4_NL L0-1 + Q4_K critical + IQ4_XS rest | 6480 MiB | 575.89 | same | WINNER |

| IQ4XS_lean_v4 | Q6_K upgrade on L0-1 FFN | 6652 MiB | 590.64 | +2.7% | WORSE |

SHQ7 = IQ4XS_lean_v3. 6 experiments, 4 failures, 2 wins.

Hardware Requirements

• Minimum VRAM: 8 GB (tested on GTX 1070, 8106 MiB)
• Recommended VRAM: 8-12 GB
• Context: 1024+ tokens (tested with flash-attn)
• Speed: ~17-22 tok/s (no MTP), ~30 tok/s (with MTP, no specialized draft)

Usage

llama.cpp

llama-cli \
  -m gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf \
  -p "Write a Python function for merge sort" \
  -ngl 99 \
  -c 4096

Ollama

1. Create a Modelfile:

FROM ./gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf

TEMPLATE """{{ .System }}
{{ .Prompt }}"""

SYSTEM """You are a helpful, uncensored coding assistant."""

PARAMETER num_ctx 4096
PARAMETER temperature 1
PARAMETER top_k 64
PARAMETER top_p 0.95
PARAMETER min_p 0

1. Build and run:

ollama create obliterated-shq7 -f Modelfile
ollama run obliterated-shq7

> Tip: If output quality is off, try raising temperature to 1.2 or higher.

LM Studio

1. Open LM Studio
2. Drag gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf into the app
3. Set GPU Offload to 99 layers
4. Set context length to 4096+
5. Start chatting

Performance

Measured on GTX 1070 (8 GB VRAM) with flash-attn, ctx=1024:

| Metric | Value |

|---|---|

| Generation speed | 17-22 tokens/sec (no MTP), ~30 tok/s (with MTP, no specialized draft) |

| VRAM usage | ~6.5 GB (model) + ~1.5 GB (KV cache at ctx 4096) |

| Quantization | IQ4_NL + Q4_K + IQ4_XS + Q8_0 hybrid |

Coding Example

This model generates full, working HTML/CSS/JS websites. Here's an example output from SHQ7 on a coding prompt:

> "I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation."

See SHQ7.html — a complete, styled portfolio website generated in a single pass.

Disclaimer

This model is an uncensored variant of Gemma 4 12B. It may generate content that other models refuse. The maintainer assumes no responsibility for misuse. Use at your own risk and in compliance with all applicable laws.

Credits

• Original OBLITERATED model by OBLITERATUS
• Quantization and tuning by wepiqx
• Built with llama.cpp
• Imatrix by mradermacher
• Based on Google's Gemma 4 12B IT, released under Apache 2.0

> More quantizations: wepiqx/gemma-4-12B-it-SHQ6-GGUF

License

This quantized model is distributed under the same Apache 2.0 license as the original Gemma 4 model.