---

license: apache-2.0

base_model:

- Qwen/Qwen3.6-35B-A3B

library_name: llama.cpp

tags:

- gguf

- moe

- quantization

- llama.cpp

- tq2_0

- quaternary

---

> Update (2026-07): now GPU-native. The TQ2_0 kernel is merged for CUDA / Metal / AMD / Vulkan — this model runs GPU-native (~144 t/s decode on an RTX 3090; ~86 t/s via expert-split on an 8 GB card; ~48 t/s on an M4 Pro via Metal). On tool-calling (BFCL) it's competitive with same-size IQ2_XXS with reasoning on (abstention 79.9% vs 84.9%). Full cross-platform + tool-call data: github.com/AgntroAI/tq2-quaternary.

Qwen3.6-35B-A3B — TQ2Q (4-level quaternary 2-bit MoE experts)

A 35B-parameter MoE that runs at reading speed on a laptop CPU — 10.85 GB, and on a

like-for-like comparison *faster and higher-quality* than a same-size community

IQ2_XXS GGUF of the same model.

Activation-aware 4-level (quaternary) 2-bit quantization of the routed expert FFNs,

packed into llama.cpp's TQ2_0 container. Attention, router, embeddings, and the output

head stay at higher precision. Runs on stock llama.cpp (CPU: AVX2 / NEON).

*TQ2Q is a human label, not a new quant type — the file is* a valid TQ2_0 GGUF and

llama-bench reports it as TQ2_0.** It distinguishes our quaternary-fit weights from a

vanilla ternary TQ2_0.

The format in one paragraph

TQ2_0 is a 2-bit container: each weight is one of 4 code words (q ∈ {0,1,2,3}),

dequantized as d·(q−1). Pure ternary only ever uses q ∈ {0,1,2} → {−d, 0, +d} and

wastes the 4th. We use it: q=3 → +2d (and a negative block scale flips the orientation),

so each 256-weight block carries {−α, 0, +α, ±2α} — a strict superset of ternary at

identical bytes and identical kernel speed. The extra level is free. Full method,

ablations, and the ternary→quaternary story: https://github.com/AgntroAI/tq2-quaternary.

Quality & speed vs a real same-size baseline

WikiText-2 vs the bf16 reference (perplexity, mean KL-divergence, top-1 agreement %);

speed is llama-bench, AVX2, 4 threads. The control is a downloadable same-size

community IQ2_XXS GGUF (Qwen_Qwen3.6-35B-A3B-IQ2_XXS.gguf, 10.67 GB).

| Variant | Size | PPL ↓ | Mean KLD ↓ | Top-1 % ↑ | Decode t/s ↑ | Prefill t/s ↑ |

|---|---|---|---|---|---|---|

| Ours — TQ2Q | 10.85 GB | 6.79 | 0.213 | 81.4 | 13.96 | 35.2 |

| Community IQ2_XXS | 10.67 GB | 7.02 | 0.246 | 79.2 | 10.71 | 22.2 |

A two-axis Pareto win — better on every quality metric and faster — against an

artifact anyone can download and check. On an Apple M4 Pro (24 GB) this decodes at

41.5 t/s. Reproduced bit-consistently across x86 AVX2, Apple NEON, and CUDA (quality

identical to ≈0.1–0.3%).

> Honest note: the raw expert-kernel advantage is ≈1.8×, but at 35B it dilutes to ≈1.3×

> because fixed costs (the 256K-vocab head, attention) grow relative to the experts. The

> more expert-dominated the model, the closer to the full 1.8×.

MMLU — downstream benchmark

The metrics above are distributional averages; MMLU shows where the 2 bits are spent.

Config: mmlu_llama, 5-shot, strict_match, full 14K-item set, served via llama-server

(/no_think, temp 0.7).

Overall: 80.58% across 61 subtasks — and the 2-bit damage is not uniform. It

concentrates in multi-step reasoning and spares factual recall:

| domain | type | TQ2Q | base ref\* |

|---|---|---|---|

| high school government | recall | 97.9 | 98.5 |

| high school biology | recall | 93.6 | 95.5 |

| miscellaneous | recall | 91.3 | 95.0 |

| professional law | reasoning | 65.7 | 70.9 |

| high school mathematics | reasoning | 57.8 | 65.2 |

| abstract algebra | reasoning | 61.0 | 70.0 |

Recall-heavy domains hold within ≈2–4 points; reasoning-heavy domains drop 5–9.

Takeaway: at 2-bit this is a strong knowledge/recall engine that gives up ground on hard

multi-step reasoning — budget accordingly.

\* Reference is a published run of the base model on this exact harness (84.76% overall). It

used a ≈4-bit-class community GGUF, not bf16, so these gaps are "2-bit vs ≈4-bit community" —

narrower than a vs-bf16 gap. temp-0.7 sampling adds run-to-run variance beyond binomial stderr.

⚠️ CRITICAL HAZARD — never requantize or merge into this GGUF

llama.cpp's quantize_row_tq2_0 (float→TQ2_0) clamps to [−1, +1], silently stripping

the 4th level and reverting the model to plain ternary — no error, quality quietly gone.

• ✅ Run it for inference (llama-completion, llama-server).
• ❌ Never llama-quantize this file, or merge a LoRA into it and re-save.

If you need a different quant, re-run the pipeline from the bf16 base model.

Run it

Recent llama.cpp with TQ2_0 CPU kernels (release b9873+). Scripted generation moved

from llama-cli to llama-completion:

llama-completion -m qwen3.6-35b-a3b-TQ2Q.gguf \
    -p "Q: A train travels 60 km in 40 minutes. What is its speed in km/h?\nA:" \
    -n 256 --temp 0 -t 4

Decode is memory-bandwidth bound — use -t 4 (P-cores) on a thin laptop. This is the

instruct model; it uses <think> reasoning blocks. qwen36-imatrix.gguf and

qwen36-bf16-wiki40.kld in this repo let you reproduce the quality numbers.

On AVX-512 / AMX CPUs (server Xeon / EPYC), add --no-repack. Stock llama.cpp's online

expert-repack silently corrupts the TQ2_0 experts there — garbage output (perplexity ≈ vocab

size), no error. AVX2 / NEON laptops are unaffected. (Confirmed on a Sapphire-Rapids box;

--no-repack restores the exact baseline.)

Files

• qwen36-fourlevel-hsw.gguf — the model (10.85 GB).
• qwen36-imatrix.gguf — per-expert importance matrix used for calibration.
• qwen36-bf16-wiki40.kld — bf16 reference logits (40×512 WikiText-2) for KLD eval.

Limitations

• Quaternary, not ternary. The 4th level is what makes it competitive; the requantize

hazard above strips it.

• Only routed expert FFNs are quantized. Non-expert layers get no benefit.
• GPU support: mainline llama.cpp has no merged TQ2_0 CUDA kernel, so there the 2-bit

experts run on CPU (full -ngl 99 offload still gives ≈4x hybrid decode). We built a

TQ2_0 CUDA kernel and MERGED it into our fork's master

(AgntroAI/llama.cpp): experts in VRAM, **≈25x

prefill** (measured on this 35B), q=3 verified correct (GPU-vs-CPU KLD 0.0078),

determinism confirmed, test-backend-ops all-green (44/44 tq2_0 cases). One master build

runs this model GPU-native across all four backends — Metal, AMD/ROCm, and Vulkan TQ2_0

kernels are merged too (see the top note). Background: https://agntro.dev/posts/tq2q-on-cuda.html

• Full negative results (gate-weighted Hessians failed; 6 proxy-vs-eval inversions) are in

the repo. Trust the eval numbers here, not intermediate proxies.

---

*Base model: Qwen/Qwen3.6-35B-A3B (Apache-2.0). This quantization inherits Apache-2.0.

Code & write-up: https://github.com/AgntroAI/tq2-quaternary*