---

license: apache-2.0

base_model:

• JetBrains/Mellum2.1-12B-A2.5B-Thinking

tags:

• gguf
• moe
• text-generation

---

Mellum2.1-12B-A2.5B-Thinking — Q3 / Q4K / Q5K GGUFs

Fast GGUF for mainstream llama.cpp, quantized antirez style: routed experts

quantized hard, everything else protected.

Recipe (antirez style)

Down-expert note: Mellum's down-expert rows are 896 wide and K-quants require

multiples of 256, so K-quant downs are impossible there (Q5_0 is llama.cpp's

standard fallback — same as bartowski's Q4_K_M layouts; the Q3_K down

falls back to Q4_0). Imatrix:

bartowski calibration v6.

| Tensor group | Q3 | Q4K | Q5K | Q5K-Q8 |

| --- | --- | --- | --- | --- |

| Routed expert gate / up | IQ3\_XXS | Q4\_K | Q5\_K | Q5\_K |

| Routed expert down | Q4\_0 | Q5\_0 | Q5\_0 | Q8\_0 |

| Shared experts, attention, output | Q8\_0 | Q8\_0 | Q8\_0 | Q8\_0 |

| Token embeddings | F16 | F16 | F16 | F16 |

| Routers, norms, biases | F32 | F32 | F32 | F32 |

| File | Size | BPW | KL ↓ | Same-top ↑ | q36 hard-smoke |

| --- | ---: | ---: | ---: | ---: | ---: |

| Mellum2.1-12B-A2.5B-Thinking-Q3-IQ3_XXS-Q4_0.gguf | 5.83 GiB | 4.12 | 0.164 | 83.6% | 9/12 |

| Mellum2.1-12B-A2.5B-Thinking-Q4K-Q5_0.gguf | 7.49 GiB | 5.30 | 0.048 | 96.6% | 9/12 |

| Mellum2.1-12B-A2.5B-Thinking-Q5K-Q5_0.gguf | 8.36 GiB | 5.91 | 0.0239 | 99.1% | 9/12 |

| Mellum2.1-12B-A2.5B-Thinking-Q5K-Q8.gguf | 9.65 GiB | 6.82 | 0.0135 | 100% | 10/12 |

KL vs Q8_0 reference logits (same prompts, 2 chunks @ ctx 60, CPU). Same

protocol put bartowski's Q5_K_M at 0.0165 (9.28 GiB), his Q6_K_S at 0.0123

(10.37 GiB) and wepiqx MERNIK-10000 at 0.0361 (9.46 GiB).

q36 hard-smoke: 12 reasoning/reading-comprehension cases from the q36 engine

test set, temp 0, llama.cpp Vulkan vs the same reference runs. For scale:

LFM2.5-24B-A2B-Q3 8/12, gemma-4-26B Q2 7/12, G9v3 Q2 5/12, Ling-3.0-tiny Q4

4/12.

Usage

# interactive chat
llama-cli -m Mellum2.1-12B-A2.5B-Thinking-Q5K-Q8.gguf

# server
llama-server -m Mellum2.1-12B-A2.5B-Thinking-Q5K-Q8.gguf --host 0.0.0.0 --port 8080

Model

Mellum2.1-12B-A2.5B-Thinking is a 64-expert MoE (12 B total, 2.5 B active)

for code reasoning. Context: 128k tokens.

Original model: JetBrains/Mellum2.1-12B-A2.5B-Thinking.

Original BF16 GGUF: bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF.

Quantized from BF16 with a streaming quantizer (bounded RAM/disk window, network

as bottleneck); the BF16 was never stored in full. License: Apache-2.0.