---

base_model: coder3101/gemma-4-12B-it-qat-q4_0-unquantized-heretic

base_model_relation: quantized

library_name: gguf

license: apache-2.0

language:

• multilingual

pipeline_tag: image-text-to-text

tags:

• gguf
• text-generation
• multimodal
• vision
• audio
• reasoning
• thinking
• agentic
• tool-use
• abliterated
• uncensored
• qat
• gemma
• q4_0
• conversational

---

gemma-4-12B-it-qat-q4_0-heretic — GGUF

This repository hosts GGUF weights and the associated vision/audio projection matrix for gemma-4-12B-it-qat-q4_0-heretic, quantized from the source floating-point tensors provided by coder3101/gemma-4-12B-it-qat-q4_0-unquantized-heretic.

About the Model

Gemma 4 12B is a Google DeepMind model from the Gemma 4 family, a multimodal (text, image, audio) architecture featuring hybrid sliding-window/global attention, configurable thinking mode, native function calling, and system-role support. This is the Quantization-Aware Training (QAT) variant, whose weights were trained to retain near-bfloat16 quality once quantized to 4-bit — unlike a naively post-training-quantized model, QAT checkpoints are specifically calibrated during training for low-bit deployment.

The heretic suffix denotes post-processing via Heretic v1.2.0 with the Arbitrary-Rank Ablation (ARA) method (row-norm preservation) performed by coder3101, which suppresses refusal behavior while preserving the model's reasoning and multimodal capabilities.

---

Model & Architecture Specifications

| Property | Value |

|---|---|

| Base Architecture | Gemma 4 12B (hybrid sliding-window/global attention) |

| Developed by | Google DeepMind |

| Primary Use | Multimodal reasoning, coding, agentic tasks |

| Context Window | 262,144 tokens |

| Supported Modalities | Text, Image, Audio |

| Vision/Audio Projector | Integrated (see repository files below) |

| Quantization-Aware Training | Yes (Q4_0-calibrated) |

| Abliteration Tool | Heretic v1.2.0 |

| Abliteration Method | Arbitrary-Rank Ablation (ARA) with row-norm preservation |

| Prompt Format | Gemma chat template (system/user/assistant roles) |

---

Abliteration Parameters

| Parameter | Value |

|---|---|

| start_layer_index | 24 |

| end_layer_index | 48 |

| preserve_good_behavior_weight | 0.3707 |

| steer_bad_behavior_weight | 0.0010 |

| overcorrect_relative_weight | 0.6177 |

| neighbor_count | 15 |

Abliteration Performance

> [!NOTE]

> The metrics below are self-reported by the original model author (coder3101) and have not been independently reproduced.

| Metric | This model | Original (google/gemma-4-12B-it-qat-q4_0-unquantized) |

|---|---|---|

| KL divergence | 0.0575 | 0 (by definition) |

| Refusals | 8/100 | 99/100 |

---

Quantization Details

| Property | Value |

|---|---|

| Text Tensors | Q4_0 (--pure, uniform quantization across all tensor types) |

| Vision/Audio Tensors | BF16 (unquantized, preserves visual/audio feature quality) |

> [!NOTE]

> Unlike standard Q4_0 builds that mix in higher-precision tensors for embeddings and select layers, this quant was produced with the --pure flag, forcing all eligible tensors to Q4_0. This most closely mirrors the QAT calibration target and is the recommended pairing for this checkpoint.

Quantization Command

./llama-quantize --pure \
  /content/model-bf16.gguf \
  /content/gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf q4_0

---

Repository Files

| Filename | Format | Size | llama.cpp Build | Description |

|---|---|---|---|---|

| gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf | Q4_0 (pure) | 6.72 GB | b9851 | Main model weights |

| mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf | BF16 | 175 MB | b9851 | Vision/audio projector — required for image or audio inputs |

---

Inference

> [!IMPORTANT]

> The vision/audio projector (--mmproj) must be supplied at runtime whenever image or audio inputs are used. Omitting it disables multimodal capability entirely.

> [!NOTE]

> Google recommends the following sampling configuration: temperature=1.0, top_p=0.95, top_k=64.

> [!NOTE]

> Thinking is enabled by including the <|think|> token at the start of the system prompt; remove it to disable reasoning output.

llama.cpp CLI (with image)

./llama-cli \
  -m gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf \
  -c 8192 \
  -ngl 99 \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --image "path/to/image.jpg" \
  -p "<start_of_turn>user\nDescribe what you see in this image.<end_of_turn>\n<start_of_turn>model\n"

OpenAI-Compatible API Server

./llama-server \
  --host 0.0.0.0 \
  --port 8080 \
  -m gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf \
  -c 16384 \
  -ngl 99 \
  --flash-attn

---

Prompt Format (Gemma Chat Template)

<start_of_turn>system
You are a helpful assistant.<end_of_turn>
<start_of_turn>user
Your question, image, or audio payload here.<end_of_turn>
<start_of_turn>model

---

Notes & Limitations

• This model is abliterated and will generate content that standard aligned models refuse. Use responsibly and in compliance with applicable laws.
• Audio input is limited to 30 seconds; video is processed as frames at up to 60 seconds (1 fps).
• Place image content before text and audio content after text in the prompt for best results.
• In multi-turn conversations, do not carry prior turns' thinking content forward — only the final response should appear in history.
• The --pure Q4_0 quantization applies uniform 4-bit precision across all eligible tensors rather than mixing precisions, which most closely matches this checkpoint's QAT calibration target.