---

library_name: gguf

pipeline_tag: text-generation

base_model: openbmb/MiniCPM5-2B

tags:

• gguf
• minicpm
• minicpm5
• imatrix
• calibrated
• reasoning
• tool-calling
• long-context
• bilingual
• local-llm
• consumer-gpu
• conversational

language:

• en
• zh

license: apache-2.0

---

MiniCPM5-2B GGUF (imatrix-calibrated)

!RACER IS OP

GGUF conversions of openbmb/MiniCPM5-2B with importance-matrix calibration, so the K-quants and i-quants keep more quality at small sizes than uncalibrated equivalents.

Who this is for: anyone who wants a 2B reasoning model with 131k context, EN/ZH bilingual chat, and native tool calling - in a 1.4-2.7 GB file that runs anywhere. Verified live: 100+ tok/s on a Tesla T4 at Q4_K_M, 114 tok/s at IQ4_XS, every quant fits in <3 GB VRAM.

> Part of RACER IS OP local-LLM drops - follow for more small fast models.

Runs on your gaming PC

Full GGUF ladder included - pick the quant that fits your card:

| Your GPU | Recommended quant | Weights | VRAM (ctx 8192, measured) | Speed (T4) |

| :--- | :--- | :--- | :--- | :--- |

| RTX 3060 / 4070 / 5070 (12 GB) | Q8_0 | 2.68 GB | 2,799 MiB | 79.8 tok/s gen |

| RTX 4060 / 3070 (8 GB) | Q6_K | 2.07 GB | 2,281 MiB | 80.8 tok/s gen |

| GTX 1660 Super / 2060 / 3050 laptop (6 GB) | Q5_K_M | 1.81 GB | 2,065 MiB | 94.7 tok/s gen |

| CPU-only / Apple Silicon / phones | Q4_K_M | 1.56 GB | 1,861 MiB | 102.6 tok/s gen |

| Smallest usable | IQ4_XS | 1.42 GB | 1,737 MiB | 114.5 tok/s gen |

Weights only; add ~1 GB for context at 8k. OOM? Drop one quant level. Headroom to spare? Go one up. On any modern laptop iGPU/CPU this is real-time chat.

Files

| File | Quant | Size | Calibrated | PPL (card) | vs F16 |

| ---- | ----- | ---- | ---------- | --- | ------ |

| MiniCPM5-2B-Q8_0.gguf | Q8_0 | 2.68 GB | - | 15.18 | +0.02 |

| MiniCPM5-2B-Q6_K.gguf | Q6_K | 2.07 GB | - | 15.24 | +0.08 |

| MiniCPM5-2B-Q5_K_M.gguf | Q5_K_M | 1.81 GB | yes | 15.39 | +0.23 |

| MiniCPM5-2B-Q4_K_M.gguf | Q4_K_M | 1.56 GB | yes | 15.65 | +0.49 |

| MiniCPM5-2B-IQ4_XS.gguf | IQ4_XS | 1.42 GB | yes | 15.88 | +0.72 |

| MiniCPM5-2B.imatrix | - | 3 MB | - | - | - |

F16 reference PPL: 15.16. PPL measured WikiText-2 test subset, 200KB, ctx 512, llama-perplexity on CPU.

Pick Q4_K_M for the best size/quality trade-off, IQ4_XS for the smallest usable file, Q8_0 for near-lossless.

Independent benchmark (measured 2026-09-20)

All numbers measured 2026-09-20 on Kaggle 2x Tesla T4 (single T4 unless noted), llama.cpp b10709, full offload -ngl 99. Method notes at the bottom.

Speed (Tesla T4, llama-bench -p 512 -n 128)

| File | Size | Prompt | Generation |

|---|---|---|---|

| MiniCPM5-2B-Q8_0.gguf | 2.68 GB | 3,334 tok/s | 79.8 tok/s |

| MiniCPM5-2B-Q6_K.gguf | 2.07 GB | 2,906 tok/s | 80.8 tok/s |

| MiniCPM5-2B-Q5_K_M.gguf | 1.81 GB | 3,002 tok/s | 94.7 tok/s |

| MiniCPM5-2B-Q4_K_M.gguf | 1.56 GB | 3,182 tok/s | 102.6 tok/s |

| MiniCPM5-2B-IQ4_XS.gguf | 1.42 GB | 3,358 tok/s | 114.5 tok/s |

Headline: 100+ tok/s on a T4 at Q4_K_M, 114 tok/s at IQ4_XS. Smaller quants decode faster (bandwidth-bound). On any modern laptop iGPU/CPU this is real-time chat.

VRAM (full offload, ctx 8192, measured nvidia-smi)

| File | VRAM |

|---|---|

| Q8_0 | 2,799 MiB |

| Q6_K | 2,281 MiB |

| Q5_K_M | 2,065 MiB |

| Q4_K_M | 1,861 MiB |

| IQ4_XS | 1,737 MiB |

Headline: every quant fits in <3 GB - runs on any GPU from the last decade, and comfortably on CPU/RAM-only machines and phones.

Quality retention (independent PPL check, WikiText-2 raw test 1.29 MB, ctx 512)

| File | PPL | Delta vs Q8_0 | Card's claimed delta vs F16 |

|---|---|---|---|

| Q8_0 | 14.16 | - | +0.02 |

| Q6_K | 14.18 | +0.02 | +0.08 |

| Q5_K_M | 14.34 | +0.17 | +0.23 |

| Q4_K_M | 14.58 | +0.42 | +0.49 |

| IQ4_XS | 14.88 | +0.72 | +0.72 |

Independent run reproduces the card's ranking and magnitudes (IQ4_XS delta matches exactly at +0.72). Q4_K_M keeps 97% of the quality signal at 58% of the Q8_0 size.

It thinks, speaks Chinese, and calls tools (all verified live)

• Reasoning model: emits [Start thinking]...[End thinking] traces (RL-trained). Thinking even happens in Chinese for Chinese prompts.
• Bilingual: exactly 3 fluent Chinese sentences about photosynthesis on demand.
• Native tool calling: with tools= passed, server returns proper OpenAI tool_calls (typed, with IDs, finish_reason: tool_calls) - get_weather{"city":"Paris"} + calculate{"expr":"237*419"} both correct. A 2B model that does tools + 131k context + EN/ZH in 1.5 GB is the download pitch.
• Spot checks (Q4_K_M): GSM8K-style arithmetic correct (29), is_prime correct with edge-case guards, exact-format instruction mostly followed.

Quant picker (verified recommendation)

• Q4_K_M (1.56 GB): best trade-off. +0.42 PPL, 103 tok/s, 1.9 GB VRAM.
• IQ4_XS (1.42 GB): smallest usable. +0.72 PPL, 115 tok/s, quality parity spot-checked (math correct).
• Q8_0 (2.68 GB): near-lossless reference.

Quickstart

# llama.cpp - server with web UI + OpenAI API
llama-server -hf saidutta69/MiniCPM5-2B-GGUF:Q4_K_M --reasoning-budget 512

# llama.cpp - CLI
llama-cli -hf saidutta69/MiniCPM5-2B-GGUF:Q4_K_M -n 1024 -p "Explain quantization in one sentence."

# Ollama
ollama run hf.co/saidutta69/MiniCPM5-2B-GGUF:Q4_K_M

# LM Studio / Jan - search "MiniCPM5-2B-GGUF", pick Q4_K_M

Also works with Docker Model Runner: docker model run hf.co/saidutta69/MiniCPM5-2B-GGUF:Q4_K_M

Usage notes

1. Give it token room: it thinks before answering. Set n_predict/max_tokens >= 1024 or answers get cut mid-thought. Server-side, --reasoning-budget 512 keeps simple tasks snappy.
2. Chat template has tool support (tokenizer.chat_template includes a tools branch) - works with llama-server/OpenAI-style tool_calls, Ollama, LM Studio.
3. 131k context inherited from base (llama.context_length = 131072 in GGUF metadata). KV is GQA (16Q/2KV, 42 layers) so long ctx stays cheap.
4. Stock llama.cpp runs everything here (standard quants, no fork needed).

How these were made

Pipeline: convert_hf_to_gguf.py --outtype f16 with latest llama.cpp, then llama-imatrix over 100 chunks of WikiText-2 train (ctx 512), then llama-quantize --imatrix for the calibrated types. The exact matrix used is included as MiniCPM5-2B.imatrix.

Method (transparency footnote)

• Throughput: llama-bench -ngl 99 -p 512 -n 128 -r 3, 1x T4 15 GB, CUDA 12.8.
• VRAM: llama-server -ngl 99 -c 8192 resident set per nvidia-smi.
• PPL: llama-perplexity -c 512 -b 512 on WikiText-2 raw test (parquet->txt, 1.29 MB). Card used a 200 KB subset, so absolutes differ; ordering and deltas match.
• Quality: llama-cli/llama-server chat, thinking-mode defaults, single runs (temp defaults) - spot checks, not a benchmark suite.
• Raw logs: VM /root/minicpm/logs/ (bench_all.log, ppl.log, m_.log, vram_.log).

License

Inherits Apache 2.0 from openbmb/MiniCPM5-2B.

Related

• MiniCPM5-2B-heretic - decensored safetensors + GGUF sibling
• RACER IS OP - Heretic Models - full collection
• Base model: openbmb/MiniCPM5-2B

> Made with love by RACER IS OP - follow for more uncensored + local-LLM models