---

license: apache-2.0

base_model: JetBrains/Mellum2-12B-A2.5B-Instruct

base_model_relation: quantized

tags:

- code-completion

- rocmfpx

- ai-max-395

- ryzen-ai-max-395

- amd

- gguf

- llama.cpp

- rocm

- gfx1151

- strix-halo

- mellum

- moe

- code

language:

- en

pipeline_tag: text-generation

---

> ### ⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

> The Mellum architecture is not merged upstream. Ignore the auto-generated

> "Use this model" commands above — use the ROCmFPX patch in patches/.

>

> 🚀 96.92 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) —

> 6.49 GiB, 1.12 GiB smaller and 7.2% faster than Q4_K_M.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to

charlie12345/ROCmFPX at commit 2809dc5,

verified with git apply --check.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: patches/README.md in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB

is configure-time, so cmake --build alone will not link them.

---

Mellum2-12B-A2.5B-Instruct — ROCmFP4 (tier 102 COHERENT) GGUF

A 4-bit ROCmFP4 quantization of

JetBrains/Mellum2-12B-A2.5B-Instruct,

built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo), with the LM head and token

embeddings held at Q6_K.

| | |

|---|---|

| File | Mellum2-12B-A2.5B-Instruct-Q4_0_ROCMFP4_COHERENT.gguf |

| Size | 6.4907 GiB (6,969,373,344 bytes) |

| BPW | 4.59 |

| ftype | Q4_0_ROCMFP4_COHERENT (102) |

| Source | BF16 GGUF (22.64 GiB) — lossless source, not a requantization |

| sha256 | 161d23aa5dd6813e348cdcbf6873beb9c1cded3b56d211379429dcaa373fc43e |

*Smaller and faster than Q4_K_M* on the target hardware — see below.

---

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

mellum is not in mainline llama.cpp. Support is open in

PR #23966 ("model: add Mellum architecture",

Xarbirus; branch Xarbirus/llama.cpp:mellum2), unmerged at time of writing. The ROCmFP4 quant

types additionally require a fork that implements them — upstream has no Q4_0_ROCMFP4_*.

⚠️ strings is not a capability check

Our build's libllama.so contained the literal string mellum and still failed with

unknown model architecture: 'mellum'. The string lives in a name table; the loader is

separate code. Grepping the binary tells you nothing — attempt the load.

---

All quant variants

All measured on one box, one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4), median of 3,

warm-up discarded — so these rows are directly comparable.

| variant | ftype | size | bpw | decode (median) | range |

|---|---|---|---|---|---|

| 4-bit COHERENT | 102 | 6.49 GiB | 4.59 | 104.99 | 104.96 – 105.73 |

| 8-bit AGENT | 115 | 11.88 GiB | 8.39 | 74.93 | 74.93 – 74.97 |

| 8-bit plain | 111 | 11.70 GiB | 8.27 | 72.76 | 72.61 – 72.79 |

Repos: 4-bit ·

8-bit AGENT ·

8-bit plain

AGENT is faster here — 74.93 vs 72.76, ranges disjoint (+3.0%). Both 8-bit builds are well below the 4-bit build's 104.99 tok/s; they exist for accuracy headroom, not speed.

> On AGENT generally: it keeps more tensors at true Q8_0 instead of the packed 8-bit type.

> That raises MTP draft acceptance on models which have an MTP head (measured +6.2% on

> Qwen3.8-27B). Mellum2 has no MTP head, so there is nothing for the extra precision to feed

> and the two 8-bit builds differ only marginally — in either direction.

Measured results

Ryzen AI MAX+ 395 (gfx1151, 128 GB unified, ROCm 7.2.4), -ngl 99 -c 4096 -fa on.

| build | size | 17×23 | capital of Japan | days in 2024 | decode |

|---|---|---|---|---|---|

| this build | 6.4907 GiB | ✅ 391 | ✅ Tokyo | ✅ 366 | 96.92 tok/s |

| Q4_K_M | 7.6063 GiB | ✅ | ✅ | ✅ | 90.37 tok/s |

| BF16 (source) | 22.6423 GiB | — | — | — | — |

+7.2% decode over Q4_K_M while 1.12 GiB smaller.

Per-tensor types (audited in the finished file, 339 tensors)

| tensor class | type |

|---|---|

| output.weight (LM head) | Q6_K |

| token_embd.weight | Q6_K |

| ffn_gate_inp router (28) | F32 |

| norms (113) | F32 |

| experts, attention projections | 4-bit |

tie_word_embeddings is false on this model, so a real output.weight exists and both

--output-tensor-type and --token-embedding-type apply. (On a tied model

--output-tensor-type is a silent no-op — worth checking before you trust it.)

Mellum2 has no shared experts and no SSM/conv state, so the protections that matter for

hybrid architectures do not apply here. Its layer_types alternate **sliding_attention ×3 →

full_attention** (n_swa = 1024), and the loader honours that pattern per layer.

---

What was NOT measured

• No perplexity run, and no quality A/B against Q4_K_M or BF16. The checks above are

memorized-fact prompts — necessary but not sufficient; a damaged model can pass them.

• No code-generation benchmark. This is a coding model and we did not evaluate it as one.
• No long-context testing (the model supports 131,072; nothing was run near it).
• No tool-calling evaluation.
• Speed figures are single measurements per build on one machine, not medians of repeated runs.

---

Model

MellumForCausalLM / mellum. 28 layers · hidden 2304 · vocab 98,304 ·

64 experts, 8 active · moe_intermediate_size 896 · sliding/full attention interval 4 ·

context 131,072 · tie_word_embeddings: false.

Base model licence: Apache-2.0 (inherited). All credit for the model itself goes to

JetBrains.