---

license: apache-2.0

base_model: JetBrains/Mellum2.1-12B-A2.5B-Thinking

library_name: gguf

pipeline_tag: text-generation

language:

- en

tags:

- gguf

- imatrix

- quantized

- moe

- thinking

- code

- agentic

---


Mellum2.1-12B-A2.5B MaxQuality iMatrix GGUF

Quantized builds of JetBrains/Mellum2.1-12B-A2.5B-Thinking — a 12B-total / 2.5B-active MoE thinking model for agentic coding, math and reasoning. Calibrated iMatrix quants, measured on real hardware, tuned launch profiles.

⚡ Speed at a glance (8K context)

| Variant | Generation speed | Max comfortable context | CPU offload |

|---|---|---|---|

| Q4_K_M (AllGPU) | ~177 tok/s short ctx · ~100 tok/s @ 40K ctx (tg, median of 30 coding tasks) | 131072 native · 262144 experimental | none — whole model resident in VRAM (≈10.5 GB) |

| Q3_K_M (AllGPU) | ~130 tok/s @ 40K ctx | 131072 | none — whole model resident in VRAM (≈9.3 GB) |

| Q3_K_M (6–8 GB VRAM) | <!-- BENCH: fill in --> | 131072 | experts of blocks 14–27 on CPU (-ot) |

<!-- BENCH: add rows for --n-cpu-moe / larger-context profiles once measured -->

Why AllGPU wins here: Mellum is a MoE (64 experts, 8 active per token). Once any experts spill to CPU, PCIe round-trips dominate — same lesson as our other MoE releases: keep the whole model in VRAM or pay dearly. Q4_K_M at 7.5 GiB fits a 16 GB card outright with the full native context in KV cache; Q3_K_M at 5.9 GiB targets cards with less VRAM to spare.

Tested hardware and software

• Linux
• NVIDIA GeForce RTX 5060 Ti, 16 GB VRAM
• AMD Ryzen 9 5950X, 16 cores / 32 threads
• 64 GB system RAM
• giveen/llama-cpp-turboquant, commit 8a891f4b (2026-07-31)

This is a MoE model (64 routed experts, 8 activated). On AllGPU profiles no -ot/--n-cpu-moe line is needed; the full-context profile below uses --n-cpu-moe to park deep experts in system RAM.

About this release

• Converted from the upstream bf16 checkpoint with the stock convert_hf_to_gguf.py; hybrid attention (SWA 1024 on 3 of every 4 layers, full attention every 4th) is handled natively by the converter — no architectural surgery.
• All files quantized with the same iMatrix pipeline as our other releases (calibration: imatrix-mellum21.dat). No per-tensor overrides — the stock K-quant presets do sensible work here.
• Thinking model: emits <think>…</think> before the final answer; use the --jinja flag and a qwen3-style reasoning parser (turboquant handles this out of the box).
• Recommended sampling (from upstream): temperature=0.6, top_p=0.95, top_k=20.

Files

| File | Size | BPW | Recipe |

|---|---|---|---|

| Mellum2.1-12B-A2.5B-Thinking_Q4_K_M_imatrix.gguf | 7.52 GiB | ~5.3 | Plain Q4_K_M + iMatrix, no manual overrides |

| Mellum2.1-12B-A2.5B-Thinking_Q3_K_M_imatrix.gguf | 5.89 GiB | ~4.2 | Plain Q3_K_M + iMatrix, no manual overrides |

Recommended launch commands

All three profiles run the whole model on the GPU (--n-cpu-moe 0 makes it explicit). TURBO_AUTO_ASYMMETRIC=0 disables turboquant's automatic asymmetric KV quantization so the symmetric turbo3/turbo4 cache types apply as specified.

131K context, single agent (recommended baseline)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q4_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 262144 --ubatch-size 4096 --cache-reuse 512

Takes a bit over 10 GB of VRAM — the whole model plus a full native-context KV cache on one card. Generation speed settles around ~100 tok/s once the context grows to ~40K tokens; short-context bursts run faster (see the speed table above). Keep it simple: no speculative decoding flags needed — this model is fastest run plain.

Two agents at once (--parallel 2)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q4_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  --ctx-size 262144 --parallel 2 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 262144 --ubatch-size 4096 --cache-reuse 512

Same ~10 GB VRAM footprint, split into two 131K slots. With two agents working simultaneously at ~5K tokens of context each, each one still gets ~100 tok/s — the practical profile for running a coding agent pair (e.g. planner + executor) off one card.

Q3_K_M — 131K context, ~9 GB VRAM

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q3_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 262144 --ubatch-size 4096 --cache-reuse 512

Fits cards with a bit under 9 GB of spare VRAM. ~130 tok/s at ~40K tokens of context — measurably faster than Q4_K_M at long context (fewer weight bytes to stream per token), and it holds the full native 131K window in VRAM.

Q3_K_M — for 6–8 GB VRAM cards (expert split CPU/GPU)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja --host 0.0.0.0 --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q3_K_M_imatrix.gguf \
  --n-gpu-layers 99 --n-cpu-moe 0 \
  -ot "blk\\.(1[4-9]|2[0-7])\\.ffn_.*_exps\\.weight=CPU" \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  --batch-size 8192 --ubatch-size 1024 --cache-reuse 2048

If your card only has 6–8 GB of VRAM, park the routed experts of the deep half (blocks 14–27) in system RAM with the -ot line while attention and the shallow layers stay on GPU. The smaller batch/ubatch and cache-reuse tune the split so prompt processing doesn't overflow the card.

256K context experiments (turbo4 KV)

TURBO_AUTO_ASYMMETRIC=0 llama-server \
  --jinja \
  --host 0.0.0.0 \
  --port 8080 \
  -m Mellum2.1-12B-A2.5B-Thinking_Q4_K_M_imatrix.gguf \
  --n-gpu-layers 99 \
  --n-cpu-moe 0 \
  --ctx-size 262144 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k turbo4 \
  --cache-type-v turbo4 \
  --batch-size 8192 \
  --ubatch-size 2048 \
  --cache-reuse 512

For experiments beyond the native 131,072 context — 262,144 tokens is 2× the model's native window, so treat output quality past 131K as uncharted territory. Uses the denser turbo4 KV cache and a smaller batch/ubatch to keep the larger cache affordable.

---

Upstream model card

This release is based on the original JetBrains/Mellum2.1-12B-A2.5B-Thinking model card, reproduced below. Original license: Apache 2.0.

> Mellum2.1 is a thinking model. Use it for complex agentic tasks, such as working in a repository, running commands, and calling tools, and for hard non-agentic problems in coding, math, and reasoning.

Mellum2.1 Highlights

Mellum2.1 is the next version of Mellum2 Thinking. The architecture is unchanged (a 12B mixture-of-experts model with 2.5B active parameters), and almost all of the work for this version went into post-training, primarily reinforcement learning (RL):

• Reinforcement learning at a new scale: RL went from a short final stage to the main part of training, after many experiments on both the methods and the data.
• More data, filtered harder: new RL tasks in math, competitive programming, science, tool use, and software engineering, combining open RL datasets with tasks JetBrains built themselves and filtering every source before training.
• Real environments for agentic skills: for software engineering, the model trains inside real repositories with a shell and file-editing tools and is rewarded when the tests pass.

As a result, Mellum2.1 handles agentic tasks much better than Mellum2. After millions of sandboxed runs in real environments during training, it explores a codebase, edits files, and checks its own changes.

Model Overview

Mellum2.1 Thinking has the following features:

• Number of Parameters: 12B total, 2.5B active
• Number of Layers: 28
• Hidden Size: 2304
• Intermediate Size: 7168
• MoE Intermediate Size: 896
• Number of Experts: 64
• Number of Activated Experts: 8
• Number of Attention Heads (GQA): 32 for Q and 4 for KV
• Context Length: 131,072
• Sliding Window: 1,024 (3 of every 4 layers)
• Vocabulary Size: 98,304
• Precision: bfloat16
• License: Apache 2.0

Evaluation

All values are percentages; higher is better except HarmBench, where lower is better. All models were evaluated by JetBrains with the same pipeline in thinking mode. All values are self-reported by JetBrains.

| Benchmark | Mellum2.1 Thinking | Mellum2 Thinking | Gemma 4 (E4B) | Qwen3.5 (9B) |

|---|---|---|---|---|

| Coding | | | | |

| LiveCodeBench v6 | 82.0 | 69.4 | 69.4 | 75.4 |

| HumanEval+ | 91.5 | 90.9 | 89.1 | 89.6 |

| MBPP+ | 79.4 | 75.4 | 70.9 | 69.8 |

| Math | | | | |

| AIME 25/26 | 83.3 | 60.1 | 45.0 | 86.7 |

| GSM-Plus | 88.3 | 87.1 | 87.4 | 91.4 |

| Agentic | | | | |

| SWE-bench Verified | 47.0 | 2.0 | 23.0 | 50.0 |

| Terminal-Bench 2.1 | 17.4 | 0.6 | 3.4 | 21.7 |

| SWE-bench Pro | 28.0 | 0.0 | 4.0 | 38.0 |

| Tool Use | | | | |

| BFCL v4 | 62.3 | 49.6 | 52.5 | 58.5 |

| WorkBench | 44.6 | 45.1 | 46.1 | 39.7 |

| ToolHop | 49.1 | 46.7 | 39.9 | 52.0 |

| Conversational | | | | |

| IFEval | 90.6 | 79.5 | 90.8 | 92.4 |

| Knowledge | | | | |

| GPQA Diamond | 64.6 | 51.0 | 53.1 | 77.8 |

| MMLU-Redux | 87.8 | 86.0 | 84.9 | 89.5 |

| MixEval-Hard | 46.4 | 41.6 | 41.4 | 50.3 |

| Safety | | | | |

| XSTest | 88.8 | 91.2 | 95.2 | 95.6 |

| HarmBench (↓) | 8.5 | 21.5 | 38.1 | 6.6 |

Notes:

• Non-agentic benchmarks use greedy decoding.
• Agentic benchmarks use the same open-source agent harness (Pi v0.73.1, shell and file tools) for every model, with each model's default sampling (temperature 1.0 for Mellum2.1), a 114K-token context, and up to 16K tokens per turn.
• AIME is the mean of AIME 2025 and AIME 2026 (30 questions each).
• BFCL v4 is the macro-average of five subtasks: v1, v2, v3, web search, memory.

Quickstart (upstream, Transformers/vLLM)

from openai import OpenAI
# Configured by environment variables
client = OpenAI()

messages = [
    {"role": "user", "content": "Find the bug in this function and explain the fix: def mean(xs): return sum(xs) / len(xs) - 1"},
]

chat_response = client.chat.completions.create(
    model="JetBrains/Mellum2.1-12B-A2.5B-Thinking",
    messages=messages,
    max_tokens=81920,
    temperature=0.6,
    top_p=0.95,
    extra_body={"top_k": 20},
)
print("Chat response:", chat_response)

License

Released under the Apache 2.0 license (inherited from the upstream model). The upstream model card and its license declaration are preserved unchanged above.