---

base_model: IFM/K2-Horizon-MoVA-36B-A4B

pipeline_tag: text-generation

library_name: gguf

license: apache-2.0

language:

- en

tags:

- k2-horizon

- 36b

- mova

- moe

- gguf

- quantized

- llama.cpp

---

K2-Horizon-MoVA-36B-A4B-GGUF

This repository contains GGUF quantizations of IFM/K2-Horizon-MoVA-36B-A4B, a sparse Mixture-of-Experts model utilizing Mixture-of-Values Attention (MoVA). It features 36B total parameters with only ~4B activated per token, enabling near-small-model inference speeds with 36B-scale intelligence.

> [!IMPORTANT]

> Architecture Support Required:

> As of early 2026, k2-horizon is a newly introduced architecture not yet merged into upstream mainline llama.cpp. To run or quantize these GGUFs, you must build from the official MBZUAI-IFM fork (model/K2Horizon branch). Running on vanilla llama.cpp will result in unknown model architecture: 'k2-horizon'.

---

Quantization Overview & Hardware Recommendations

| File Name | Quant | Size | Recommended VRAM (Full Offload) | Minimum System RAM (CPU) | Description |

| :--- | :--- | :--- | :--- | :--- | :--- |

| K2-Horizon-MoVA-36B-A4B-Q3_K_S.gguf | Q3_K_S | 16.4 GB | 20 GB (RTX 3090/4090) | 24 GB | Smallest footprint; lowest memory usage. |

| K2-Horizon-MoVA-36B-A4B-Q3_K_M.gguf | Q3_K_M | 17.7 GB | 22 GB (RTX 3090/4090) | 28 GB | Balanced 3-bit quantization. |

| K2-Horizon-MoVA-36B-A4B-Q4_K_S.gguf | Q4_K_S | 21.4 GB | 24 GB (RTX 3090/4090) | 32 GB | Great performance/size ratio for 24 GB GPUs. |

| K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf | Q4_K_M | 22.4 GB | 24 GB+ (RTX 3090/4090) | 32 GB | Recommended balance of quality and memory efficiency. |

| K2-Horizon-MoVA-36B-A4B-Q5_K_M.gguf | Q5_K_M | 26.4 GB | 32 GB | 36 GB | Near-lossless retention; high precision. |

| K2-Horizon-MoVA-36B-A4B-Q6_K.gguf | Q6_K | 30.8 GB | 36 GB | 40 GB | High precision; preserves subtle router logits. |

| K2-Horizon-MoVA-36B-A4B-Q8_0.gguf | Q8_0 | 39.8 GB | 48 GB (2× 24 GB / A6000) | 48 GB | Maximum precision; near identical to original BF16. |

---

Installation & Setup

1. Build llama.cpp with K2-Horizon Support

# Clone the official architecture fork
git clone -b model/K2Horizon [https://github.com/MBZUAI-IFM/llama.cpp.git](https://github.com/MBZUAI-IFM/llama.cpp.git)
cd llama.cpp

# Build with hardware acceleration (enable CUDA if running on NVIDIA GPUs)
cmake -B build -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release # Add -DGGML_CUDA=ON for NVIDIA GPU
cmake --build build -j$(nproc) --target llama-cli llama-server

Quickstart Guide

Running via llama-cli

./llama.cpp/build/bin/llama-cli \
  -m ./K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --temp 1.0 \
  --top-p 0.95 \
  -p "<|ifm|im_start|>user\nWrite an efficient Python script for asynchronous data fetching.<|ifm|im_end|>\n<|ifm|im_start|>assistant\n"

Prompt Format & Reasoning Traces

K2-Horizon uses ChatML-style delimiters with native thinking tags:

<|ifm|im_start|>system
You are a helpful assistant.<|ifm|im_end|>
<|ifm|im_start|>user
Your query goes here.<|ifm|im_end|>
<|ifm|im_start|>assistant
<ifm|think>
[Model generates intermediate reasoning steps here]
</ifm|think>
[Final output generated here]<|ifm|im_end|>