---

license: apache-2.0

base_model: IFM/K2-Horizon-MoVA-36B-A4B

base_model_relation: quantized

library_name: llama.cpp

pipeline_tag: text-generation

language:

• en

tags:

• gguf
• llama.cpp
• k2-horizon
• mova
• moe
• long-context
• quantized
• text-generation

---

K2-Horizon-MoVA-36B-A4B GGUF

Community GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B.

<div align="center" style="background-color:#f59e0b;color:#ffffff;padding:16px 20px;border-radius:10px;line-height:1.7;">

☕ If this GGUF made your day easier, a coffee would make mine.<br>

<a href="https://ko-fi.com/ngquocvinh" style="color:#ffffff;"><strong style="color:#ffffff;">Send a coffee ☕</strong></a><br>

I build and test these releases myself. Your coffee helps keep me going.<br>

Thank you for supporting this work.

</div>

About K2-Horizon-MoVA-36B-A4B

K2-Horizon-MoVA-36B-A4B

is a sparse Mixture-of-Experts model with Mixture-of-Values attention (MoVA).

The upstream checkpoint has 36B total parameters and activates about 4B

parameters per token. It advertises a native 524,288-token context window and

is intended for reasoning, coding, agentic, and long-context workloads. See the

official model card for

the original model, serving recipes, prompt conventions, and upstream results.


*Upstream K2-Horizon-MoVA-36B-A4B benchmark results; the image and scores are

from the official model card.*

This is a quantization-only release. No training, fine-tuning, merging, or

model capability modification other than GGUF conversion and quantization was

performed here.

Files

All files below were produced independently from the same locked BF16 GGUF

input. The labels describe memory/quality trade-offs, not a guarantee for every

task.

| Quantization | Intended profile |

|---|---|

| Q8_0 | Highest-bit published profile; reference option. |

| Q6_K | High-fidelity local profile. |

| Q5_K_M | Recommended general-purpose profile. |

| Q4_K_M | Balanced size/quality profile. |

| IQ4_NL | Importance-aware 4-bit profile. |

| IQ4_XS | Smaller importance-aware 4-bit profile. |

| Q3_K_L | 3-bit profile with a larger K variant. |

| Q3_K_M | Compact 3-bit profile. |

| IQ3_M | Importance-aware 3-bit profile. |

| IQ3_S | Smaller importance-aware 3-bit profile. |

| Q2_K | Very small 2-bit profile; validate on your workload. |

| Q2_K_S | Smaller 2-bit profile; validate on your workload. |

| IQ2_XS | Extra-small importance-aware profile; experimental. |

| IQ1_M | Extreme compression profile; experimental. |

| Q1_0 | Extreme compression profile; experimental. |

Recommended quantized profiles

These are three practical starting points, all excluding the BF16 reference:

| Profile | Choose it when |

|---|---|

| K2-Horizon-MoVA-36B-A4B-Q8_0.gguf | You want the closest practical profile to the BF16 reference. |

| K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf | You want the main size/quality balance. |

| K2-Horizon-MoVA-36B-A4B-IQ3_M.gguf | You need a smaller importance-aware profile. |

Fidelity measurements

The table below compares every published GGUF file against the

BF16 reference on a held-out WikiText pilot: eight chunks from wiki.test.raw and eight chunks from

wiki.valid.raw, using a 4,096-token context and the same K2 llama.cpp runtime.

Values are averaged across the two splits. Lower Mean KLD, ΔPPL, and RMS Δp,

and higher Top-1 agreement, indicate closer next-token behavior to BF16. The

BF16 reference mean PPL was 6.815612 in this pilot. These are next-token

fidelity measurements, not a direct percentage of capabilities retained and not

a substitute for task-specific evaluation.

| File | Mean KLD (↓) | Top-1 vs BF16 (↑) | ΔPPL | RMS Δp |

|---|---:|---:|---:|---:|

| K2-Horizon-MoVA-36B-A4B-Q8_0.gguf | 0.004015 | 97.270% | +0.071% | 1.822% |

| K2-Horizon-MoVA-36B-A4B-Q6_K.gguf | 0.005739 | 96.281% | +0.180% | 2.191% |

| K2-Horizon-MoVA-36B-A4B-Q5_K_M.gguf | 0.010129 | 95.185% | +0.527% | 2.865% |

| K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf | 0.017449 | 93.564% | +0.855% | 3.702% |

| K2-Horizon-MoVA-36B-A4B-IQ4_NL.gguf | 0.019676 | 93.350% | +1.055% | 3.909% |

| K2-Horizon-MoVA-36B-A4B-IQ4_XS.gguf | 0.019835 | 93.558% | +0.957% | 3.909% |

| K2-Horizon-MoVA-36B-A4B-Q3_K_L.gguf | 0.047255 | 89.686% | +2.760% | 5.925% |

| K2-Horizon-MoVA-36B-A4B-Q3_K_M.gguf | 0.051186 | 89.265% | +2.920% | 6.213% |

| K2-Horizon-MoVA-36B-A4B-IQ3_M.gguf | 0.054490 | 89.066% | +4.063% | 6.483% |

| K2-Horizon-MoVA-36B-A4B-IQ3_S.gguf | 0.055718 | 89.060% | +4.034% | 6.482% |

| K2-Horizon-MoVA-36B-A4B-Q2_K.gguf | 0.116123 | 84.810% | +9.201% | 9.529% |

| K2-Horizon-MoVA-36B-A4B-Q2_K_S.gguf | 0.159896 | 82.334% | +13.465% | 11.167% |

| K2-Horizon-MoVA-36B-A4B-IQ2_XS.gguf | 0.236500 | 77.873% | +20.785% | 13.879% |

| K2-Horizon-MoVA-36B-A4B-IQ1_M.gguf | 0.604200 | 66.142% | +69.948% | 22.709% |

| K2-Horizon-MoVA-36B-A4B-Q1_0.gguf | 10.019445 | 1.478% | +2,069,546.308% | 59.829% |

In this pilot, Q1_0 shows very large divergence from the BF16 reference;

the extreme-compression profiles should therefore be treated as experimental.

The machine-readable results are available in

reproducibility/quality-summary.tsv,

with corpus hashes, evaluation settings, and runtime provenance in

reproducibility/manifest.md.

Quick start

Use a K2-Horizon-capable llama.cpp build. The explicit partial offload and

CPU MoE options in this example keep the model within the available accelerator

memory when full offload is not possible; adjust the GPU-layer count for your

hardware.

./llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-Q6_K.gguf \
  --chat-template-file reproducibility/chat_template_smoke_user.jinja \
  --jinja \
  --reasoning off \
  -p 'Answer briefly in English: What is GGUF, and why is it useful for running language models locally?' \
  -n 128 -c 4096 \
  --device CUDA0 \
  --gpu-layers 8 \
  --cpu-moe

The included template is the compatible single-turn template used by the

release smoke test. The upstream full tool-aware Jinja template is not claimed

as certified by this package. Long-context use may require substantially more

memory than the short example above.

Reproducibility and validation

The GGUF files were quantized directly from the locked upstream BF16 input. Q8_0

was generated without an imatrix; the other ladder members used the

model-specific combined importance matrix. Every published file passed the

load/generate smoke test before upload. Runtime throughput data, when present,

is supplementary and is recorded separately from the BF16 fidelity comparison

above in reproducibility/runtime-summary.tsv.

The public package includes compact reproduction inputs and scripts. Raw

conversion, imatrix, quantization, smoke-test, fidelity, and benchmark logs are

kept locally under reports/ and are intentionally not uploaded.

Checksums for all published artifacts and public reproduction files are in

SHA256SUMS.txt. The locked source revision, BF16 checksum,

runtime commit, calibration inputs, and validation settings are in

reproducibility/manifest.md.

License and attribution

The upstream model is released under the Apache License 2.0. Preserve upstream

attribution and the included LICENSE when redistributing these

derivative artifacts.

These are community GGUF quantizations, not an official IFM release or

endorsement.