---

license: apache-2.0

base_model: JetBrains/Mellum2-12B-A2.5B-Thinking

tags:

- gguf

- quantization

- apex

- imatrix

- moe

- code

- mellum

pretty_name: Mellum2-12B-A2.5B-Thinking APEX GGUF

---

Mellum2-12B-A2.5B-Thinking — APEX GGUF

Measured-allocation GGUF quants of JetBrains/Mellum2-12B-A2.5B-Thinking (12B total / 2.5B active MoE, 28 layers, 64 experts, thinking, Apache-2.0).

Calibrated with a code-weighted imatrix (~70% multi-language code, 336 chunks) and evaluated by KL divergence against the bf16 model, not perplexity alone.

Why this model needs deliberate allocation

Mellum2's moe_intermediate_size is 896, and 896 is not divisible by 256. Every K-quant type (Q6_K/Q5_K/Q4_K/Q3_K/IQ4_XS/IQ3_/IQ2_) requires rows divisible by 256, so ffn_down_exps — 3.70B params, 30.4% of the model — cannot take a K-quant at all. Everything else is fine (hidden 2304, FFN 7168, vocab 98304).

Stock Q4_K_M handles this by falling back per tensor. Its mix wants Q6_K on every other layer's ffn_down; unable to, it falls upward to Q8_0:

| | stock Q4_K_M |

|---|---|

| ffn_down_exps | 14 × Q8_0 (8.5 bpw) + 14 × Q5_0 |

| file | 8.07 GB |

| effective | 5.31 bpw vs 4.85 nominal (+9.5%) |

So a third of the model receives bits by fallback rather than by choice, while the K-quantable 70% sits at Q4_K. These quants set ffn_down_exps explicitly instead.

Tiers

| tier | file | size | bpw | ffn_down_exps |

|---|---|---|---|---|

| i-quality | Mellum2-12B-APEX-i-quality-imat.gguf | 9.72 GB | 6.40 | Q5_1 |

| compact | Mellum2-12B-APEX-compact-imat.gguf | 8.40 GB | 5.53 | Q5_0 |

| mini | Mellum2-12B-APEX-mini-imat.gguf | 6.92 GB | 4.55 | IQ4_NL |

Measured quality — KL divergence vs bf16

Paired against bf16 logits over 100 chunks of a code-heavy corpus (no wikitext contamination).

| model | size | bpw | Mean KLD ↓ | 99% KLD ↓ | same top-p ↑ |

|---|---|---|---|---|---|

| i-quality | 9.72 GB | 6.40 | — | 0.391 | 97.58 ± 0.10 % |

| compact | 8.40 GB | 5.53 | 0.0264 ± 0.0009 | 0.562 | 97.16 ± 0.10 % |

| mini | 6.92 GB | 4.55 | 0.0722 ± 0.0022 | 1.579 | 95.21 ± 0.13 % |

| stock Q4_K_M | 8.07 GB | 5.31 | 0.0633 ± 0.0019 | 1.347 | 95.62 ± 0.13 % |

compact is 2.4× closer to bf16 than stock Q4_K_M for 0.22 bpw more (0.33 GB) — mean KLD 0.0264 vs 0.0633, and +1.54 points of top-token agreement.

mini is the honest trade in the other direction: 14% smaller than stock Q4_K_M and measurably further from bf16. It is the smallest sensible tier for this model — below IQ4_NL the only legal step for that 30% is Q2_0 at 2.25 bpw, so there is a cliff rather than a slope.

Choosing

• compact — the recommended default. Best fidelity per byte here by a wide margin.
• i-quality — when you want the closest match to bf16.
• mini — when 6.92 GB is the constraint and you accept the quality cost.

Notes

• Perplexity was too blunt to separate these: every difference sat inside its ±0.019 error bars, and on one comparison PPL ranked the models in the opposite order to KL divergence. PPL scores only the observed token's probability; KLD scores the whole distribution. The KLD numbers above are the ones to trust.
• Requires a llama.cpp build with LLM_ARCH_MELLUM support.