---

license: apache-2.0

base_model: empero-ai/Qwythos-9B-v2

tags: [gguf, qwen35, imatrix, quantized, kld]

language: [en]

---

!banner

Qwythos-9B-v2 — full GGUF ladder · measured KLD fidelity + loop-rate test

> This is NOT my model. Weights and training are by empero-ai (Qwythos-9B-v2, Apache-2.0). This is an independent GGUF repack whose value-add is what the other community GGUFs don't publish: an objective, measured fidelity sweep (KLD / PPL / Top-1 vs the Q8 reference) plus a real loop-rate benchmark on the quantized weights. No weights were modified.

What v2 fixes (per empero-ai): eliminated looping (6.7% → 0% at greedy decoding), restored the MTP head, cleaner identity, FTPO-trained. This repack is built from the official MTP-Q8_0 so the MTP head is preserved.

✅ Recommended files

| Use case | File | Size | Top-1 vs Q8 |

|---|---|---|---|

| Archival / best | Qwythos-9B-v2-Q8_0.gguf | 9.5 GB | 100% |

| Near-lossless | Qwythos-9B-v2-Q6_K.gguf | 7.6 GB | 98.2% |

| Balanced | Qwythos-9B-v2-Q5_K_M.gguf | 6.6 GB | 97.2% |

| Best compact (imatrix) | Qwythos-9B-v2-IQ4_XS.gguf | 5.3 GB | 96.2% |

| 6 GB VRAM (imatrix 3-bit) | Qwythos-9B-v2-IQ3_M.gguf | 4.5 GB | 93.0% |

| Smallest (lossy) | Qwythos-9B-v2-Q3_K_M.gguf | 4.7 GB | 90.8% |

> The imatrix wins at every bit level. IQ4_XS dominates Q4_K_M (5.3 vs 5.8 GB, 96.2% vs 94.6% Top-1) and IQ3_M dominates Q3_K_M (4.5 vs 4.7 GB, 93.0% vs 90.8% Top-1, KLD 0.065 vs 0.102). For a 6 GB VRAM card, IQ3_M is the sweet spot — smaller and more faithful than plain Q3_K_M, with room left for context. Prefer the IQ tiers over the plain Q tiers at 3- and 4-bit.

📊 Measured fidelity — vs the Q8_0 reference

llama-perplexity --kl-divergence over 24×512-token chunks. KLD (Kullback–Leibler divergence) is the gold-standard quant-fidelity metric — it measures how far the quant's full next-token distribution drifts from the reference, catching damage that PPL and Top-1 alone miss. Top-1 match = how often the quant's argmax equals the reference's.

| Quant | Size | PPL vs Q8 | KLD mean | KLD p50 | Top-1 match |

|---|---|---|---|---|---|

| Q8_0 | 9.79 GB | ref | ref | ref | 100.0% |

| Q6_K | 7.56 GB | +0.41% | 0.0070 | 0.0008 | 98.2% |

| Q5_K_M | 6.64 GB | +1.20% | 0.0151 | 0.0020 | 97.2% |

| Q4_K_M | 5.78 GB | +1.60% | 0.0377 | 0.0067 | 94.6% |

| IQ4_XS | 5.33 GB | +1.10% | 0.0243 | 0.0045 | 96.2% |

| IQ3_M | 4.52 GB | +5.63% | 0.0652 | 0.0210 | 93.0% |

| Q3_K_M | 4.74 GB | +6.90% | 0.1017 | 0.0272 | 90.8% |

!fidelity vs size

> Reference = Q8_0, not BF16. Qwythos-v2 is a hybrid qwen35 arch (3:1 Gated-DeltaNet linear-attention : full attention + MTP), which llama.cpp does not lower cleanly from safetensors — so the ladder is requantized from empero-ai's official Q8_0 (near-lossless) with --allow-requantize + an imatrix, and fidelity is measured against that Q8_0.

🧪 Real loop-rate benchmark (measured on the quant)

v2's headline is "looping 6.7% → 0% at greedy decoding." We verified it on the quantized weights (Q5_K_M): greedy decode (temp 0), 10 diverse prompts × 220 tokens, flag any generation that degenerates into a repeated cycle or collapsed vocabulary.

Result: 0.0% loop rate (0/10 prompts) — confirms the quantization preserves v2's loop-fix — greedy (temp 0), 300 tokens each, mean repetition 0.038, every generation coherent (852–1527 chars incl. a naturally-repetitive counting prompt). (Method + raw per-prompt output in scripts/loop_bench.py + reports/loop_bench.json.)

🚀 Run it

ollama run hf.co/KikoCis/Qwythos-9B-v2-Full-Ladder-KLD-GGUF:Q5_K_M
llama-server -m Qwythos-9B-v2-Q5_K_M.gguf -c 32768

Sampling: qwen3.5-family reasoning models want temp 1.0 / top_p 0.95 / top_k 20. For Ollama use RENDERER qwen3.5 + PARSER qwen3.5 (recent build). v2 no longer loops at greedy, but temp 1.0 is still recommended for reasoning quality.

⚠️ Honest scope

Two things are measured here: (1) KLD/PPL/Top-1 fidelity vs Q8, and (2) a loop-rate test. This is not an independent capability benchmark — for agentic/reasoning scores (MMLU, GSM8K, HumanEval, GPQA), see empero-ai's card (their numbers, from their internal harness, not re-verified here).

🔁 Provenance

Requantized from the official MTP-Q8_0 with llama-quantize --allow-requantize + imatrix. Scripts in scripts/ (pipeline_v2.sh, parse_metrics.py, loop_bench.py), per-quant KLD logs in reports/, SHA-256 in reports/artifact-sha256sums.txt.

Importance matrix (qwythos-v2.imatrix, shipped in this repo): computed fresh on this model's own Q8 (not reused from other weights), over a general-purpose calibration corpus — a ~250 KB mix of general English prose and code (standard for imatrix generation; not weight-specific data). Reuse it directly for your own quants. Note: the imatrix does not cover the MTP/nextn head (block 32) — a normal forward pass doesn't activate it — so IQ3_XS bails ("missing importance matrix … very low-bit"); IQ3_M quantizes cleanly (it tolerates the uncovered head) and is the recommended 3-bit tier.

Credit

Model © empero-ai (Apache-2.0). This repack + measured metrics: KikoCis.