---

license: other

base_model: kai-os/Grug-12B

tags:

• gguf
• tq3
• gemma4
• quantization
• llama-cpp
• blackwell
• cuda

base_model_relation: quantized

quantized_by: maczzzzzz (via turbo-tan/llama.cpp-tq3)

---

Grug-12B TQ3_4S — GGUF

TQ3_4S quant of kai-os/Grug-12B — a compact-reasoning fine-tune of google/gemma-4-12B-it, produced via turbo-tan's llama.cpp-tq3 CUDA fork. Benchmarked on a Blackwell RTX 5060 Ti (16 GB). Context scaling is comfortable: 152k at q4_0 KV with ~55 t/s throughput and 8+ GB VRAM headroom.

Grug-12B is a specialized reasoning model that was trained to compress chain-of-thought while maintaining reasoning quality — shorter reasoning traces without losing correctness on math and code tasks. It was converted from the original safetensors (no pre-quant GGUF existed) using convert_hf_to_gguf.py and quantized in a single step.

The big caveat: this compact-reasoning fine-tune comes at a real cost on instruction-following benchmarks. The model over-reasons on every prompt — responses are 12-15k chars even for simple formatting tasks — which depresses IFEval scores significantly vs. the base Gemma-4-12B-it. Use this quant where reasoning quality and trace compression matter, not where you need clean, concise output on the first shot.

File

| File | Size | Quant | BPW |

|---|---|---|---|

| Grug-12B-TQ3_4S.gguf | 5.9 GB | TQ3_4S (turbo-tan) | ~3.8 bpw |

NOT a stock llama.cpp quant

TQ3_4S (TurboQuant 3-bit 4-state) is a custom weight format unique to turbo-tan/llama.cpp-tq3. Stock llama.cpp and nixpkgs llama-cpp will exit with unknown quantization at load time. Use the llama-server/llama-cli from the tq3 fork.

Scope of these benchmarks — read this first

These numbers are a light baseline, not a thorough quant evaluation. The bench suite is built for production agent workload regression-detection, not for the kind of multi-axis sweep that upstream quant maintainers typically publish. Specifically:

• Harness scope is bounded. The numbers come from llama-bench (KV context ladder) and lm-eval-harness IFEval n=50. That's a regression suite, not a quality benchmark.
• Sample sizes are small. Throughput is single-rep on a single GPU. IFEval is n=50. None are powered for multi-seed significance.
• No perplexity / wikitext / MMLU / GSM8K. GSM8K runs were initiated but hit transient network issues. The compact-reasoning fine-tune prioritizes reasoning-token compression, not academic benchmarks.
• Single GPU class (Blackwell 16 GB). All measurements on an RTX 5060 Ti 16 GB, CUDA 13.2. Cross-hardware generalization is NOT implied. The companion ROCmFP4_FAST quant for AMD RDNA4 is in a separate repo.
• No human eval. "55 t/s throughput and IFEval 0.14" is not a quality verdict on reasoning quality.

What this IS good for: a quick signal that the quant (a) loads, (b) runs at high throughput, (c) has comfortable VRAM headroom, (d) scales to full 152k context. What this is NOT good for: claiming "this is the best quant of this model," reproducing academic benchmark results, or substituting for upstream's validation work.

For a rigorous view, see kai-os/Grug-12B (parent model), google/gemma-4-12B-it (base model), and the raw bench reports attached as BENCH-*.md in this repo.

What we measured

Context ceiling (q4_0 KV, Blackwell RTX 5060 Ti)

The 12B footprint gives ample VRAM headroom. Throughput is flat across the entire ladder.

| KV type | Ctx | TG tok/s | PP tok/s | VRAM |

|---|---|---|---:|---:|---:|

| f16 | 32k | 56.9 | 47.6 | 7.1 GB |

| q8_0 | 64k | 54.7 | 47.3 | 6.9 GB |

| q4_0 | 64k | 54.6 | 47.3 | 6.6 GB |

| q4_0 | 98k | 54.7 | 47.4 | 6.7 GB |

| q4_0 | 131k | 54.7 | 47.4 | 6.9 GB |

| q4_0 | 152k | 54.7 | 47.2 | 7.0 GB |

VRAM headroom at 152k: >8 GB. No VRAM constraints on this card class.

IFEval n=50 — reasoning-channel analysis

Grug's compact-reasoning fine-tune produces dense reasoning traces that bloat every response (12-15k chars each). This significantly hurts instruction-following benchmarks.

| Metric | Grug-12B TQ3_4S | google/gemma-4-12B-it (Q5_K_XL, baseline) |

|---|---|---|

| prompt_level_loose | 0.14 ± 0.05 | 0.34 |

| inst_level_loose | 0.28 | ~0.49 |

The compact-reasoning fine-tune costs ~20pp on IFEval compared to the base model. This is expected — Grug was trained on math/code reasoning compression, not instruction-following. The model excels at producing compact-but-complete reasoning traces; use it for agentic pipelines where reasoning density matters more than output format compliance.

Practical tip: serve with --reasoning-format none so the output lands in the content field rather than reasoning_content — required for any downstream tooling that reads content. This flag instructs the server to strip the thinking/reasoning channel and emit only the final answer.

Quick start

# Build turbo-tan's tq3 fork
git clone https://github.com/turbo-tan/llama.cpp-tq3
cd llama.cpp-tq3
mkdir build && cd build
cmake .. -DGGML_CUDA=ON -DCUDA_DOCKER_ARCH=sm_120
make -j$(nproc)

# Serve
./bin/llama-server \
  -m /path/to/Grug-12B-TQ3_4S.gguf \
  --port 8081 \
  -ngl 99 \
  -c 32768 \
  --reasoning-format none

Reproduce the quant

# From kai-os/Grug-12B safetensors, converted to BF16 GGUF first:
python3 convert_hf_to_gguf.py /path/to/kai-os/Grug-12B --outfile Grug-12B-BF16.gguf

# Then quantize:
/path/to/llama-quantize \
  --allow-requantize \
  Grug-12B-BF16.gguf \
  Grug-12B-TQ3_4S.gguf \
  TQ3_4S

Files in this repo

| File | Purpose |

|---|---|

| Grug-12B-TQ3_4S.gguf | The quantized model (LFS-tracked, 5.9 GB) |

| README.md | This file |

| BENCH-*.md | Raw bench data and cross-format context-ceiling reports |

What's NOT in this repo (caveats)

• Stock llama.cpp will not load this file. TQ3_4S is a custom weight format unique to turbo-tan/llama.cpp-tq3.
• Grug is NOT a general-purpose instruct model. It's a specialized compact-reasoning model. Use for math, code reasoning, and agentic pipelines where reasoning-token compression matters. Do NOT use for instruction-following tasks (IFEval, formatting) or direct chat where clean output is expected — the reasoning bloat will hurt quality.
• No CUDA non-Blackwell bench. Tested on RTX 5060 Ti (sm_120) only.
• 256K native ctx not tested. The parent Gemma-4-12B-it supports 256K native ctx. Tested up to 152k successfully; higher contexts should work on larger VRAM.
• No vision/multimodal test. This variant is text-only.
• No MTP / speculative-decode bench. Not applicable on this architecture.
• No quality benchmark (perplexity, MMLU, GSM8K). The fine-tune optimizes for reasoning-token compression; academic benchmarks may not reflect its strengths or weaknesses.

Provenance

• Source model: kai-os/Grug-12B (license: other — verify commercial use terms)
• Base model: google/gemma-4-12B-it (Gemma 4 license)
• Conversion: convert_hf_to_gguf.py (BF16 GGUF from 7 sharded safetensors, 23.8 GB → TQ3_4S 5.9 GB)
• Quantizer: turbo-tan/llama.cpp-tq3 commit 77dd77473
• Build hardware: Node D — Blackwell RTX 5060 Ti 16 GB, CUDA 13.2, NixOS
• Bench harnesses: llama-bench (context ladder), lm-eval-harness (IFEval n=50)

License

This quant is derived from kai-os/Grug-12B (license: other — verify commercial use is permitted before deployment), which is a fine-tune of google/gemma-4-12B-it (Gemma 4 license). The TQ3_4S quant format is provided by turbo-tan/llama.cpp-tq3 (MIT).