---

license: apache-2.0

base_model:

- openbmb/MiniCPM5-2B

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- quantized

- llama-cpp

- ollama

- llama

- minicpm

- minicpm5

- q8_0

- kld

- tool-calling

- reasoning

- long-context

- on-device

- conversational

language:

- en

- zh

---

!banner

<div style="border:2px solid currentColor; font-family:ui-monospace,'SF Mono','Cascadia Mono',Consolas,monospace;">

<div style="border-bottom:1px solid currentColor; padding:6px 12px; font-size:11px; letter-spacing:3px; text-transform:uppercase; opacity:0.7; text-align:center;">KIKOCIS // 8-BIT GGUF // MEASURED AGAINST BF16</div>

<div style="padding:14px; display:flex; flex-wrap:wrap; align-items:center; justify-content:center; gap:18px;">

<pre style="margin:0; font-size:9px; line-height:1.15;">

BF16 ──────────────┐

token_embd ── bf16 │ KLD 0.000616

blk.0..41 ── q8_0 ├──▶ top-1 98.67 %

output ── bf16 │ PPL +0.0109

───────────────────┘

</pre>

<div style="text-align:center;">

<div style="font-size:23px; font-weight:800; letter-spacing:1px;">MiniCPM5-2B · Q8_0-XL</div>

<div style="font-size:12.5px; letter-spacing:1px; opacity:0.8; margin-top:5px;"><span style="white-space:nowrap;">llama · 2B</span> · <span style="white-space:nowrap;">128K context</span> · <span style="white-space:nowrap;">−11 % KLD vs plain Q8_0</span></div>

</div>

</div>

<table style="display:table; table-layout:fixed; width:100%; margin:0; border-collapse:collapse; font-size:12px;">

<tr>

<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">FORMAT</div><div style="font-weight:700;">GGUF</div></td>

<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">SIZE</div><div style="font-weight:700;">3.18 / 2.68 GB</div></td>

<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">ARCH</div><div style="font-weight:700;">llama, 42 layers</div></td>

<td style="border-top:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">CONTEXT</div><div style="font-weight:700;">131,072</div></td>

</tr>

<tr>

<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">QUANTS</div><div style="font-weight:700;">Q8_0-XL · Q8_0</div></td>

<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">VALIDATION</div><div style="font-weight:700;">KLD / PPL / top-1</div></td>

<td style="border-top:1px solid currentColor; border-right:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">RUNS ON</div><div style="font-weight:700;">llama.cpp · Ollama</div></td>

<td style="border-top:1px solid currentColor; padding:8px 12px;"><div style="font-size:10px; letter-spacing:1px; opacity:0.6;">LICENSE</div><div style="font-weight:700;">Apache-2.0</div></td>

</tr>

</table>

</div>

MiniCPM5-2B — Q8_0-XL + Q8_0 GGUF

> 8-bit GGUFs of OpenBMB's MiniCPM5-2B, each one measured against the BF16 original. Q8_0-XL keeps the token embeddings and the output head in BF16 and puts every transformer block in Q8_0: 11 % lower KLD and 26 % less perplexity drift than a plain Q8_0, for +0.5 GB. Runs in ~3–4 GB of RAM at 8K context.

>

> This is OpenBMB's model, our quantization. OpenBMB also publishes their own GGUFs; we measured their Q8_0 alongside ours (table below) and it is equivalent to our plain Q8_0. What this repo adds is the XL variant and the numbers.

✅ Recommended files

| Use case | File | Notes |

|---|---|---|

| Best fidelity at 8 bits | MiniCPM5-2B-Q8_0-XL.gguf | Lowest KLD and PPL drift here. Recommended. |

| Smallest 8-bit | MiniCPM5-2B-Q8_0.gguf | Standard Q8_0, 0.5 GB lighter. |

📦 Files

| Quant | Bits (blocks / embed + head) | File size |

|---|---|---:|

| Q8_0-XL | 8 / 16 | 3.18 GB |

| Q8_0 | 8 / 8 | 2.68 GB |

The embedding table and output head are unusually large for a 2B model (130,560-token vocabulary × 2,048 = 1 GB in BF16 between the two), which is why keeping them in BF16 costs 0.5 GB and is where most of the remaining 8-bit error lives.

📊 Metrics — fidelity vs the BF16 reference

KLD (Kullback–Leibler divergence, nats): how far the quant's next-token distribution drifts from BF16. Top-1 = how often the quant's most likely token is the same as BF16's. PPL Δ = perplexity above BF16 (11.196).

| File | Size GB | PPL | PPL Δ | KLD mean | KLD median | KLD p95 | KLD p99 | Top-1 match |

|---|---:|---:|---:|---:|---:|---:|---:|---:|

| BF16 reference | 5.04 | 11.196 | 0 | 0 | 0 | 0 | 0 | 100 % |

| Q8_0-XL | 3.18 | 11.207 | +0.0109 | 0.000616 | 0.000375 | 0.00193 | 0.00444 | 98.67 % |

| Q8_0 | 2.68 | 11.211 | +0.0147 | 0.000692 | 0.000451 | 0.00209 | 0.00463 | 98.57 % |

| Q8_0 — OpenBMB's official file, for reference | 2.68 | 11.211 | +0.0147 | 0.000692 | 0.000450 | 0.00209 | 0.00464 | 98.56 % |

Standard errors: ±0.000005 on mean KLD, ±0.003 on PPL Δ, ±0.05 points on top-1. The XL gap is well outside them.

_{Machine-readable: metrics/quant-summary-with-kld.json / .csv. Per-file detail in reports/.}

📈 Chart

!fidelity vs size

🧮 Will it fit?

Memory ≈ file size + KV cache. MiniCPM5-2B uses grouped-query attention (2 KV heads), so long context is cheap:

| Context | KV cache (F16) | Q8_0-XL total | Q8_0 total |

|---|---:|---:|---:|

| 8K | 0.35 GB | ~3.6 GB | ~3.1 GB |

| 32K | 1.4 GB | ~4.6 GB | ~4.1 GB |

| 128K (native) | 5.6 GB | ~8.9 GB | ~8.4 GB |

🚀 How to run it

# llama.cpp — the GGUF carries OpenBMB's chat template (tools + reasoning)
llama-server -m MiniCPM5-2B-Q8_0-XL.gguf -c 32768 --jinja --temp 1.0 --top-p 0.95

# Ollama — Modelfiles for 8K / 32K / 128K are included
ollama run hf.co/KikoCis/MiniCPM5-2B-GGUF:Q8_0-XL
ollama create minicpm5 -f Modelfile-32k

Sampling: temperature 1.0, top_p 0.95 — OpenBMB's own defaults (generation_config.json).

Reasoning: the model thinks before answering; Ollama returns that in a separate thinking field, llama-server in reasoning_content with --jinja.

Tools: native function calling through the embedded chat template — use the tools parameter of the OpenAI-compatible API.

⚠️ Good to know

• At 8 bits both files are already very close to BF16; the XL gain is real but small in absolute terms. Pick XL if you have 0.5 GB to spare, plain Q8_0 otherwise.
• No weights were changed: this is a faithful quantization of OpenBMB's release.
• We have not run agentic or task benchmarks on these files — the numbers above measure fidelity to the original model, not its capability. For capability, see OpenBMB's card.

📐 Evaluation methodology

• What: KLD, perplexity and top-1 agreement of each GGUF against the BF16 GGUF converted from the same weights.
• Data: wikitext-2 test split (wiki.test.raw), first 64 chunks of 2,048 tokens.
• Tool: llama-perplexity --kl-divergence-base (BF16 logits) then --kl-divergence per file.
• Source: openbmb/MiniCPM5-2B at revision f97400052a43d642bbc6e9975e2397e3ae6a6b52.
• Load test: both files loaded and answered correctly in llama.cpp and Ollama (chat template, reasoning split).
• Date: 2026-10-10.

🔁 Provenance & reproducibility

• Exact commands: scripts/build.sh — download at the pinned revision → BF16 + Q8_0 convert → Q8_0-XL quantize → KLD/PPL.
• Checksums: reports/artifact-sha256sums.txt — shasum -a 256 -c to verify.
• Per-file results: reports/, metrics/.

🗒️ Changelog

• 2026-10-10 — v1: Q8_0-XL and Q8_0, with KLD/PPL/top-1 vs BF16 and OpenBMB's Q8_0 as a reference row.

📚 Credit & license

Model, weights and training: © OpenBMB — openbmb/MiniCPM5-2B. Quantization, measurements and Modelfiles: KikoCis. Apache-2.0, same as upstream.