---

license: apache-2.0

base_model:

- allenai/OLMoE-1B-7B-0924

library_name: llama.cpp

tags: [gguf, moe, quantization, llama.cpp, tq2_0, quaternary]

---

> Update (2026-07): now GPU-native. The TQ2_0 kernel is merged for CUDA / Metal / AMD / Vulkan — OLMoE runs GPU-native (~384 t/s decode on an RTX 3070 Ti; ~165 t/s on an M4 Pro; ~79 t/s on an Intel Arc 140V iGPU via Vulkan). Full cross-platform speed table: github.com/AgntroAI/tq2-quaternary.

OLMoE-1B-7B — TQ2Q (4-level quaternary 2-bit MoE experts)

Activation-aware 4-level (quaternary) 2-bit quantization of OLMoE-1B-7B's routed

expert FFNs, in llama.cpp's TQ2_0 container. At 2.06 GB it **beats the same-size

IQ2_XXS bar on quality AND runs ≈1.8× faster** on the same CPU.

*TQ2Q is a human label, not a new quant type — the file is* a valid TQ2_0 GGUF and

llama-bench reports it as TQ2_0.** It distinguishes our quaternary-fit weights from a

vanilla ternary TQ2_0. TQ2_0 is a 2-bit container (4 code words, dequant d·(q−1));

pure ternary uses only 3 and wastes q=3. We use it (q=3 → +2d), so each block carries

{−α, 0, +α, ±2α} — a strict superset of ternary at identical bytes and kernel speed.

Full method & the ternary→quaternary story: https://github.com/AgntroAI/tq2-quaternary

Files

| File | What it is |

|---|---|

| olmoe-1b-7b-TQ2Q.gguf (2.06 GB) | The flagship — pure GPTQ-4level-hsw fit (11.03 ppl) |

| olmoe-1b-7b-TQ2Q-rs.gguf (2.06 GB) | Same experts + recalibrated F32 routers — the base the adapter pairs with |

| olmoe-1b-7b-TQ2Q-R16-adapter.gguf (302 MB) | Rank-16 sidecar --lora adapter (runtime only, never merge) |

| olmoe-0924-imatrix.gguf (22 MB) | Per-expert calibration imatrix (wiki.train, 100 chunks) |

| olmoe-0924-bf16-wiki40.kld (1.03 GB) | bf16 reference logits — reproduce our KLD numbers exactly |

The -rs base and the adapter were fit together (router recalibrated against the

sidecar-corrected model); the 10.81 config is -rs + adapter. Use the plain flagship if

you don't want the adapter.

Quality & speed (WikiText-2 vs bf16 reference; llama-bench, AVX2 -t 4)

| Variant | Size | PPL ↓ | KLD ↓ | Top-1 % ↑ | Decode t/s (AVX2) | Decode t/s (M4) |

|---|---|---|---|---|---|---|

| bf16 reference | 13.8 GB | 8.28 | — | 100 | — | — |

| IQ2_XXS (the bar) | 2.06 GB | 11.28 | 0.301 | 75.2 | 26.1 | 109 |

| Ours — TQ2Q | 2.06 GB | 11.03 | 0.300 | 74.7 | 47.7 | 191 |

| Ours — +sidecars+router | 2.36 GB | 10.81 | 0.277 | 75.8 | ≈47 | ≈185 |

| TQ2_0 RTN (collapse control) | 2.06 GB | 4726 | 6.31 | 12.8 | — | — |

At equal bytes ours beats IQ2_XXS on ppl and KLD and runs ≈1.8× faster (47.7 vs

26.1 t/s AVX2; 191 vs 109 on Apple M4). RTN ternary at the same bytes collapses to

gibberish — the fit, not the container, is what matters. The +sidecars+router variant

trades 0.30 GB (a runtime --lora adapter) for a further quality bump on all three

metrics. Reproduced bit-consistently across x86 AVX2, Apple NEON, and CUDA.

⚠️ CRITICAL HAZARD — never requantize or merge into this GGUF

llama.cpp's quantize_row_tq2_0 (float→TQ2_0) clamps to [−1, +1], silently stripping

the 4th level and reverting to plain ternary — no error, quality gone. Run it for

inference; never llama-quantize it or merge a LoRA into it and re-save. Apply the

sidecar adapter with --lora (runtime, not merged). Need a different quant? Re-run from

the bf16 base.

Run it

llama-completion -m olmoe-1b-7b-TQ2Q.gguf \
    -p "The key insight of information theory is" -n 128 --temp 0 -t 4

# optional: the sidecar recovery adapter (runtime, NOT merged) — pairs with the -rs base
llama-completion -m olmoe-1b-7b-TQ2Q-rs.gguf --lora olmoe-1b-7b-TQ2Q-R16-adapter.gguf \
    -p "The capital of Lithuania is" -n 128 --temp 0 -t 4

Recent llama.cpp (b9873+); scripted generation is llama-completion, not llama-cli.

Decode is memory-bandwidth bound — -t 4 (P-cores) on a thin laptop.

On AVX-512 / AMX CPUs, add --no-repack — stock llama.cpp's online expert-repack corrupts

TQ2_0 MoE experts there (silent garbage, no error; the same TQ2_0 repack path we confirmed on

our larger TQ2Q models). AVX2 / NEON unaffected.

Limitations

Quaternary, not ternary (the 4th level is the point; the hazard strips it). Only routed

expert FFNs are quantized. OLMoE's dominance is scale-contingent — at 35B (see the Qwen3.6

repo) it's a Pareto win, not a knockout. Honest negatives (gate-weighted Hessians failed; 6

proxy-vs-eval inversions) are in the GitHub repo.

GPU support: mainline llama.cpp has no merged TQ2_0 CUDA kernel, so there the 2-bit

experts run on CPU (full -ngl 99 offload still gives ≈4x hybrid decode). We built a

TQ2_0 CUDA kernel and MERGED it into our fork's master

(AgntroAI/llama.cpp): experts in VRAM, ≈25x

prefill (measured on the 35B), q=3 verified correct (GPU-vs-CPU KLD 0.0078), determinism

confirmed, test-backend-ops all-green (44/44 tq2_0 cases). One master build runs this

model GPU-native across all four backends (Metal, AMD/ROCm, and Vulkan kernels are merged too —

see the top note). Background: https://agntro.dev/posts/tq2q-on-cuda.html

---

*Base model allenai/OLMoE-1B-7B-0924 (Apache-2.0); this quant inherits Apache-2.0.

"TQ2Q" is a human label; shipped grid is quaternary {−α,0,+α,±2α} in a standard TQ2_0

GGUF. Code: https://github.com/AgntroAI/tq2-quaternary*