---

license: apache-2.0

base_model: IFM/K2-Horizon-MoVA-36B-A4B

pipeline_tag: text-generation

library_name: gguf

tags:

• k2-horizon
• mova
• moe
• llama.cpp
• gguf
• k-quants

---

K2-Horizon-MoVA-36B-A4B GGUF quants


Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo

GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B (MoVA mixture-of-value attention, 100 experts / 8 active, 512K context).

Converted with the official k2-official llama.cpp branch (MBZUAI-IFM port, commit 35999d101).

Files

| File | Quant | Size |

|---|---|---|

| k2horizon-q3_k_m.gguf | Q3_K_M | ~16.9 GB |

| k2horizon-q4_k_s.gguf | Q4_K_S | ~18.7 GB |

| k2horizon-q4_k_m.gguf | Q4_K_M | ~20.8 GB |

| k2horizon-q5_k_m.gguf | Q5_K_M | ~25.4 GB |

| k2horizon-q8_0.gguf | Q8_0 | ~39 GB |

Requirements

Use a llama.cpp build from the k2-official branch of MBZUAI-IFM/llama.cpp (or anything that merges that port). Mainline llama.cpp does NOT support the k2-horizon architecture.

Usage

llama-server -m k2horizon-q4_k_m.gguf -ngl 99 --n-cpu-moe 42 -c 8192 -b 4096 -ub 2048 -t 32

On consumer GPUs (12 GB VRAM class), keep attention on GPU and let expert weights spill to CPU/RAM via --n-cpu-moe N. Lower N = more expert layers on GPU = more VRAM used, faster decode. Tune N until your VRAM budget is respected. -ub 2048 matters for prefill throughput on the CPU-offload path.

Notes

• Plain K-quants from F16, no imatrix.
• All quants verified loading and generating on RTX 3060 12GB with the offload config above.