---

base_model: inclusionAI/Ling-3.0-tiny

license: mit

pipeline_tag: text-generation

tags:

- gguf

- moe

- quantized

- antirez-recipe

---

antirez-recipe quants

MoE quants following the antirez recipe:

routed experts quantized hard, everything else protected.

| Tensor class | Quant | Notes |

| --- | --- | --- |

| blk..ffn_gate_exps, blk..ffn_up_exps (ffn_gate_up_exps if fused) | Q5_K (imatrix) | routed-expert gate/up |

| blk.*.ffn_down_exps | Q8_0 (imatrix) | routed-expert down (K-quant for quality) |

| blk..ffn_{gate,up,down}_shexp, all attn_, output.weight | Q8_0 | shared experts / attention / head |

| token_embd.weight | F16 | embeddings |

| router (ffn_gate_inp), norms, sinks, biases | source precision (BF16/F32) | never quantized by llama.cpp |

Base type Q8_0 + --tensor-type overrides, imatrix calibration

(see SPEC per-model imatrix links). Runs on upstream llama.cpp — no fork or

patch needed.

Validation per file: gguf-dump tensor audit (experts = GU/DOWN, protection intact),

llama-cli smoke load, llama-perplexity spot ppl + --kl-divergence vs BF16 logits.

Files

| File | Size | SHA-256 |

| --- | ---: | --- |

| Ling-3.0-tiny-Q4K-MXFP4-Q8.gguf | 5091030464 | 54859c79a6700c944b686bade6ef3ec062f1ca06c31a32fdeac8d2b350c870b4 |

| Ling-3.0-tiny-Q5_K_M.gguf | 6898091232 | 34bd66526660e6d9c02b0f592cd2c0a95fc17b24b9345900d4d95061a8f18d07 |

Imatrix: https://huggingface.co/bloomer010/Ling-3.0-tiny-GGUF/Ling-3.0-tiny-imatrix.gguf

Recipe

Antirez recipe on inclusionAI/Ling-3.0-tiny (BF16 source: inclusionAI/Ling-3.0-tiny-GGUF, Ling-3.0-tiny-bf16.gguf):

• ffn_gate_exps / ffn_up_exps -> Q5_K
• ffn_down_exps -> Q8_0
• attention + shared experts + output -> Q8_0
• token embeddings -> F16
• imatrix: bloomer010/Ling-3.0-tiny-GGUF (Ling-3.0-tiny-imatrix.gguf, sha-verified identical locally)

15,065 MiB BF16 -> 6,572 MiB (6.98 BPW).

Tokenizer note: Ling-3.0-tiny ships its own chat template (tokenizer.ggml.pre = bailingmoe2) — it works as-is on upstream llama.cpp, no extra flags needed.

Validation (llama.cpp, Ling arch)

• 12-case hard-smoke eval (temp 0, ctx 15360, full-resident -ngl 999, AMD Vulkan/UMA): 4/12 PASS, ~120-124 t/s. Fails 2826, 10360, 6034, 10567, 9487, 1606, 1716, cwe121 (long-search loops that never commit a final answer; generations cut mid-thought). Passes 6500, 1618 and both big-code cases. Evaluated both raw and with the model's own template: generations are identical either way (4/12 both) — the template contributes nothing on this model.
• Exact output tokens (own tokenizer) over the 4 solved: 26,017 = 6,504 tok/solved.
• KL divergence vs the official inclusionAI/Ling-3.0-tiny-GGUF Q8_0 (ds4 short prompts, ctx 60, 2 chunks, CPU): per-chunk KL 0.00995 / 0.01375 (avg 0.0119), delta-p RMS 3.3-3.8%, same-token top-1 96.6 / 94.8%.
• Eval parity with the Q4K-MXFP4-Q8 row (4/12 @ 124 t/s): this Q5_K_M build is a strictly higher-fidelity artifact of the same model (KL 0.0119, recipe-best class) at the same smoke score.