---

license: apache-2.0

base_model: IFM/K2-Horizon-MoVA-36B-A4B

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- moe

- apex

- quantized

- imatrix

- k2-horizon

- mova

- llama.cpp

---

K2-Horizon-MoVA-36B-A4B — APEX GGUF

Imatrix-guided APEX quantization of

IFM/K2-Horizon-MoVA-36B-A4B — MBZUAI's

Institute of Foundation Models (the LLM360/K2 lineage), released 2026-09-01, Apache-2.0.

Not related to Moonshot's Kimi K2. Coincidental name collision: MBZUAI's K2 lineage (K2-65B

with LLM360, 2024) predates Kimi K2 and shares nothing architecturally — no MLA here, and MoVA

does not exist in Kimi K2.

> ### ⚠️ Requires a forked llama.cpp — upstream CANNOT load this model

> Stock llama.cpp will fail to load these files. As of 2026-09-04 upstream has zero support

> for this architecture (0 hits for k2_horizon / K2Horizon / mova at origin/master

> 64a155d24). The base model's card claims "PR to llama.cpp is in progress"; **no such PR

> exists** on GitHub.

>

> You need this fork and branch:

> ### 👉 MBZUAI-IFM/llama.cpp — branch model/K2Horizon

>

> ```bash

> git clone -b model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp llama.cpp-k2horizon

> cd llama.cpp-k2horizon

> cmake -B build -DGGML_CUDA=ON # drop -DGGML_CUDA=ON for a CPU-only build

> cmake --build build -j --config Release

> ```

>

> Verified at commit 35999d101 (build b10671) — five commits on top of upstream, a textbook

> new-arch addition (conversion, hparams/tensor loading, compute graph, graph adjustment +

> tokenizers, chat template). It is 145 commits behind upstream, and almost nothing relevant is

> missing: tools/imatrix 0 commits, tools/perplexity 0, src/llama-quant.cpp 1 (a threading

> fix, not a correctness one).

>

> Every llama-server / llama-perplexity / llama-quantize invocation on this page means the

> binary from that build tree.

Architecture — and why it needed special handling

arch k2-horizon 798 tensors, 37.445 B params

layers 48, first 3 dense

FFN MoE 100 experts, top-8, +1 shared; expert ffn 768; sigmoid gating, scale 2.5

MoVA 64 VALUE experts, top-4, on 45 of 48 layers

attention 32 heads / 8 KV, head_dim 128

context 524,288 vocab 250,624

MoVA — Mixture-of-Values Attention — is the unusual part. The attention values are a

64-expert routed bank (attn_v_exps [2560, 1024, 64], 7.55 B params = 20.2% of the model),

gated by attn_v_gate [2560, 64].

That gate is the reason this quant exists in this form. attn_v_gate is a router, and the

shape says so — 64 output columns for 64 experts, structurally identical to ffn_gate_inp

[2560, 100] for 100 FFN experts. Its logits pick experts by top-k, so weight error flips a

discrete choice rather than adding smooth noise.

The trap: llama.cpp arch-refuses ffn_gate_inp by name, but does not know this name — so a

recipe request is honoured and the router really does ship quantized. Left to a generic

attn_* rule it lands on a Q3_K floor. Here it is pinned to F32, verified in the shipped

bytes (attn_v_gate types: {F32: 45}, quantised-low count 0), at a cost of 24.2 MiB on a

21 GiB file.

*Generalizable lesson: any architecture that names its router something new is unprotected by

name-based handling. This is the second such case we've hit, after Flash-Next's

per_layer_token_embd.*

Block legality is completely clean — every mass-carrying ne0 (2560, 1024, 768, 4096, 6144) is

256-divisible, so no QK_K type is illegal and no block-32 substitution occurs.

Files

| tier | size | bpw | wikitext-2 PPL | vs bf16 | agentic frontier-9 |

|---|---:|---:|---:|---:|---|

| i-quality | 21.05 GiB | 4.83 | 7.0252 ± 0.04550 | +1.31% | 9/9 · 142/142 |

| compact | 16.69 GiB | 3.83 | 7.0657 ± 0.04604 | +1.90% | 9/9 · 142/142 |

| mini | 14.86 GiB | 3.41 | 7.1900 ± 0.04696 | +3.69% | 9/9 · 142/142 |

| bf16 reference | 69.78 GiB | 16.0 | 6.9341 ± 0.04515 | — | (not hosted) |

Compression 3.31× / 4.18× / 4.70×. All three PPL arms were measured back-to-back on the same

binary, same corpus and the same offload config, so the tier-to-tier steps below are directly

comparable rather than assembled from separate sessions.

Which tier to take — compact is the value pick

The step costs are very unequal, and the agentic suite cannot see this at all (all three tiers score

142/142). Perplexity can:

| step | Δ PPL | Δ% | significance | disk saved | cost per GiB |

|---|---:|---:|---:|---:|---:|

| i-quality → compact | +0.0405 | +0.58% | 0.63 σ | 4.36 GiB | 0.13 %/GiB |

| compact → mini | +0.1243 | +1.76% | 1.89 σ | 1.83 GiB | 0.96 %/GiB |

compact buys 4.36 GiB for a perplexity change that is not statistically resolvable (0.63 σ).

Going the next rung to mini costs 7× more perplexity per GiB saved for less than half the

space. So:

• compact (16.69 GiB) — take this unless you need the last 1.8 GiB. Best value by a wide margin.
• i-quality (21.05 GiB) — take it if you have the RAM and want the smallest measurable loss.
• mini (14.86 GiB) — only when 1.8 GiB genuinely decides whether the model runs. Its +3.69%

vs bf16 is the first tier here whose loss is clearly resolvable (3.93 σ), i.e. the IQ2_S middle

band does cost something real even though every test still passes.

For scale on what "resolvable" means here: re-measuring the same file twice on this box moves PPL

by 0.0018, so the compact→mini step (+0.1243) is ~69× the measurement noise floor and is a real

effect, whereas i-quality's own +1.31% against bf16 sits at 1.42 σ and does not exclude zero.

How the three tiers differ

All three pin the MoVA router (attn_v_gate) at F32 and keep ffn_gate_inp protected. The budget

difference lands almost entirely on the routed expert bands:

| tier | routed edge (5.3 B) | routed near (7.6 B) | routed middle (21.2 B) |

|---|---|---|---|

| i-quality | Q6_K | Q6_K | Q3_K |

| compact | Q4_K | IQ4_XS | IQ3_XXS |

| mini | Q4_K | Q3_K | IQ2_S |

mini runs the middle band — 57% of the model — at 2.5 bpw and still passes every test case.

That is MoE redundancy working as advertised: 100 routed experts at top-8 absorb 2-bit noise that

destroys a small dense model at the same width. But passing is the entry bar, and perplexity shows

the 2-bit band is not free: mini is the only tier here whose loss against bf16 is clearly

resolvable. If you need smaller, the next rung down (IQ2_XXS, ~2.06 bpw) is where we stopped —

untested, and given how sharply the cost rose from IQ3_XXS to IQ2_S, not recommended blind.

compact and mini were built from a 310-chunk code-inclusive imatrix (uploaded as

k2-horizon-36b-calib2.imatrix) rather than i-quality's 120-chunk prose one, because calibration

corpus choice is known to move routed experts much more than attention — which is exactly what

these tiers quantise hardest. At i-quality's 4.83 bpw the same swap measured as a null (limitation 3).

Note that i-quality carries a coarser allocation than the smaller tiers: its budget lands on

Q6_K/Q6_K/Q3_K with nothing in between, while compact and mini spread smoothly. See limitation 1.

Read the confidence interval, not the point estimate

delta +0.0929 PPL (+1.34%)

combined sigma 0.0641 -> 1.45 sigma

95% CI [-0.47%, +3.15%]

The CI includes zero, so at 137 chunks this corpus cannot resolve the quantization cost.

That is not a null result — quantization always costs something — it means the cost is small

relative to the measurement noise floor. Tightening it needs a larger eval corpus, not more

--chunks: 137 × 2048 = 280,576 tokens is everything wiki.test.raw yields with this

tokenizer.

This delta is measured against our own bf16 — same box, same binary, same corpus, same

tokenizer, no borrowed baseline and no cross-machine term. That is only possible because the

bf16 fits in host RAM (69.78 GiB), which it does not for most models this family's size.

Corpus pinned: WikiText-2 raw test split, 1,290,590 bytes,

sha256 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08.

Do not compare 7.0270 to other models' PPL. This tokenizer (vocab 250,624) needs 137 chunks

where Qwen's needs 145 on the identical corpus; higher per-token perplexity is the arithmetic

consequence of packing more information per token, not a quality signal. The percentage cost

is the more portable quantity.

Agentic capability — 42/42 tasks, 253/253 test cases

Real agentic coding suites: the model drives an actual tool-use loop (read/write/run files) and is

graded by a fresh pytest it never sees, at temp 1.0. Full sweep on a Xeon + RTX 4060 Ti 16 GB:

| suite | tasks | test cases | output tokens | wall |

|---|---|---|---:|---:|

| general (25) | 25/25 | 25/25 | 15,965 | 1.06 h |

| hard (5) | 5/5 | 59/59 | 25,402 | 1.05 h |

| frontier (9) | 9/9 | 142/142 | 65,370 | 2.63 h |

| repair/mutation (3) | 3/3 | 27/27 | 2,353 | 0.16 h |

| TOTAL | 42/42 | 253/253 | 109,090 | 4.90 h |

Perfect score, zero failures, zero context truncations. Single-slot throughout

(n_slots=1, kv_unified=false, n_ctx_slot=49152, verified in the server log, not assumed).

All three tiers, frontier-9 (the hardest suite)

| tier | tasks | test cases | output tokens | wall |

|---|---|---|---:|---:|

| i-quality | 9/9 | 142/142 | 65,370 | 158 min |

| compact | 9/9 | 142/142 | 36,270 | 78 min |

| mini | 9/9 | 142/142 | 44,578 | 97 min |

A full score here is the entry bar, not the result — this suite is built to saturate for a

competent coder, so 142/142 means the tier qualified, and it is not evidence that dropping from

4.83 to 3.41 bpw costs nothing. The discriminating metric at equal score is output tokens, and at

one run per tier it cannot separate them: across these four runs of near-identical models,

btree_insert_delete alone spans 1,731 to 79,583 tokens (46×) and six of nine tasks span more

than 4×. So the token column is published as absolute per-run numbers, and **no ratio between tiers

should be read off it.** Separating the tiers on cost would need >=3 reps each.

An earlier frontier-9 run on a DGX Spark (GB10) scored 8/9 (139/142) — the single difference was

one flip-prone task where that run hit a DP reconstruction off-by-one. Same model, same quant,

same commit. That is run-to-run variance, not a finding about either box, and it is the same

effect the token spread above shows.

Running it

> ### ⚠️ These commands need the FORKED binary — stock llama.cpp cannot load these files

> If you scrolled straight here to copy a command, read this first. k2_horizon / mova has

> zero support in upstream llama.cpp, so a stock llama-server will refuse the file at the

> architecture check. Build the fork:

>

> ```bash

> git clone -b model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp llama.cpp-k2horizon

> cd llama.cpp-k2horizon && cmake -B build -DGGML_CUDA=ON && cmake --build build -j

> ```

>

> Full detail, verified commit and what the branch changes: see the top of this card.

> Every llama-server, llama-perplexity and llama-quantize invocation below means that

> binary, which is why the path is spelled out rather than assumed to be on your PATH.

./llama.cpp-k2horizon/build/bin/llama-server \

-m K2-Horizon-MoVA-36B-A4B-APEX-i-quality.gguf \

--ctx-size 32768 --parallel 1 \

-ngl 99 -ncmoe 40 \

--cache-type-k q8_0 --cache-type-v q8_0 \

-fa on --jinja \

--reasoning-format deepseek --reasoning on --reasoning-effort high

--n-cpu-moe does not move the MoVA bank. It knows FFN experts only, so attn_v_exps stays

on the GPU unless you ask with -ot "attn_v_exps=CPU". Counter-intuitively the *default is the

fast one*: keeping that bank on the card is worth +30% generation (13.5 vs 10.4 t/s), and at

this quant it is 4.06 GiB and fits — at bf16 it was 14.06 GiB and had to be offloaded.

Generation speed on a 16 GiB RTX 4060 Ti + 20 CPU threads (-ncmoe N = layers whose FFN

experts stay on CPU, so 48 = all of them):

| config | gen t/s | prompt t/s | peak VRAM |

|---|---:|---:|---:|

| -ngl 0 (CPU only) | 5.6 | 16.4 | — |

| -ncmoe 48 -ot attn_v_exps=CPU | 10.4 | 21.9 | — |

| -ncmoe 48 (MoVA on GPU) | 13.5 | 24.0 | — |

| -ncmoe 40 | 15.4 | 28.4 | — |

| -ncmoe 32 | 17.5–18.3 | 32.3 | 15,665 MiB |

| -ncmoe ≤30 | OOM | — | — |

-ncmoe 32 is fastest but not the one to run — 719 MiB of headroom on a 16,384 MiB card,

and ~2 GiB of what is already there is the desktop. Anything that grows the desktop's footprint

mid-run turns a working config into a cudaMalloc failure. -ncmoe 40 trades ~2 t/s for ~3 GiB

of slack. Identical invocations measured 18.3 then 17.5 t/s, so treat one-decimal differences as

noise. These are single-stream, empty-cache figures.

KV cache is 192 KiB/token at f16 (48 × 8 × 128 ÷ 256) — 24 GiB at 128k, **96 GiB at the

advertised 512k**, so the full context is not reachable on a workstation. Use q8_0 KV.

It is a reasoning model. The template uses <ifm|think>; llama-server's deepseek extractor

handles the non-standard tag correctly (reasoning_content populated, content clean, no leak).

Reasoning at effort=high is heavy — 168 tokens for "17*23" — so give it a generous output

budget or you get empty content with finish_reason=length.

Honest limitations

Three, all specific and all fixable — recorded because a quant card that only lists strengths

is not useful.

1. The allocation is a cliff, not a gradient. The histogram is

{Q8_0: 144, Q6_K: 265, F32: 45, Q3_K: 112} — nothing between Q6_K and Q3_K. The middle

expert band (21.2 B params, 57% of the model) dropped five menu steps in one move while

12.9 B stayed at Q6_K, because structural allocation steps whole bands and one step less

would have overshot the target. A measured per-tensor allocation would smooth this; this

architecture does not have a sensitivity name-map yet. (Allocator accuracy itself was good:

predicted 22.59 GB, artifact 22.601 GB.)

1. attn_v_exps placement is reasoned by analogy, not measured. It is treated as a routed

expert bank on its access pattern (sparse 4-of-64), which is the defensible default. But a

KDA census found value tensors the most sensitive attention projection, and those were a

single dense V seeing every token where each expert here sees ~1/16 of traffic. **At 20.2% of

the model this deserves a sensitivity pass before anyone calls it measured.**

1. The calibration is thin, and for MoVA a bigger corpus does not fix it. This tier was

built from a 120-chunk imatrix (61,440 tokens) on a general/scientific-prose corpus, because

that corpus — not --chunks — was the binding limit.

A second imatrix was then built on a 2.6× larger, code-inclusive corpus (310 chunks,

158,720 tokens). Comparing the two directly, on the accumulated .counts in each file:

| | FFN experts (ffn_{up,down}_exps) | MoVA experts (attn_v_exps) |

|---|---|---|

| 120-chunk prose | 0 unrouted, min 1 route | 6 unrouted of 2,880 |

| 310-chunk mixed | 0 unrouted, min 3 routes | the same 6 unrouted |

The unrouted set is identical — blk.{35,39,40,41,42,44}.attn_v_exps, experts

60/15/41/24/10/45 — across two corpora that differ by a whole modality (the first contains no

code at all). And there is no gradual tail behind them: those 6 cells are exactly 0 while the

1st percentile of all others is 114 routes and the median is 7,274. Only 6 of 2,880 cells sit

below 10 routes, and they are the same 6.

So these read as structurally unselected experts, not a sampling gap. Stated as measured:

neither corpus ever routes them. Both are English-dominant, so a genuinely different input

distribution has not been ruled out. Six of 2,880 cells (0.2%) have no importance data.

And the larger corpus does not improve perplexity either — measured. A second tier was

built from the 310-chunk imatrix with a byte-identical allocation (same recipe, same type

histogram), so calibration data was the only variable:

| arm | PPL |

|---|---|

| this file (120-chunk imatrix) | 7.0252 +/- 0.04550 |

| rebuild (310-chunk imatrix) | 7.0269 +/- 0.04550 |

+0.0017, i.e. 0.026 sigma — indistinguishable, and nominally the wrong direction. The

control that makes it conclusive: re-measuring this same file moved it 0.0018 from its own

earlier 7.0270, so the same-file noise floor is 1.06x the effect being tested. (Likely

mechanism: MoVA's top-4 routing turns a 1-ULP difference into a different expert selection.)

The rebuild also cleared all 42 agentic tasks (253/253), which is the suites' entry bar rather

than a null result, and its perplexity is inside the same-file noise floor. It is not published

as a separate tier on that basis. (Tokens-at-equal-score, the discriminating agentic metric, was

not run to sufficient reps to separate them either way.)

**Consequence: calibration is empirically ruled out as the lever at this size point, which

promotes limitation 1 (the allocation cliff) to the prime suspect for the +1.34%.**

Provenance

Built with quantkit dynamic_apex.py; the MoVA

classifier rules landed as 9e91f50. Verified before building: 13 assertions against the real

tensor list pass, and a regression over **294 name/layer combinations from other supported

architectures shows zero movement** — the rules anchor on attn_v_gate.weight$ /

attn_v_exps.weight$, which no other architecture has.

PPL measured on the fork binary for both arms, which makes any kernel offset common-mode and

cancels it in the delta.