---

license: apache-2.0

library_name: llama.cpp

pipeline_tag: text-generation

tags:

- gguf

- moe

- k2-horizon

- mova

- llama.cpp

---

K2-Horizon-MoVA-36B-A4B GGUF quants

GGUF quants of IFM/K2-Horizon-MoVA-36B-A4B.

The base model has 36B total parameters and approximately 4B active parameters per token.

These are straight llama-quantize conversions from the original BF16 GGUF published by IFM.

No importance matrix or calibration data was used.

Files

| File | Quantization | Size |

|---|---:|---:|

| K2-Horizon-MoVA-36B-A4B-Q4_K_M.gguf | Q4_K_M | 22.37 GB |

| K2-Horizon-MoVA-36B-A4B-Q5_K_M.gguf | Q5_K_M | 26.44 GB |

| K2-Horizon-MoVA-36B-A4B-Q6_K.gguf | Q6_K | 30.77 GB |

Required llama.cpp build

K2 Horizon architecture support is required. Use the IFM fork/branch until support is

available in the upstream version you are using:

git clone --depth 1 --single-branch --branch model/K2Horizon \
  https://github.com/MBZUAI-IFM/llama.cpp.git llama.cpp-k2horizon
cd llama.cpp-k2horizon
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j"$(nproc)"

Run a quant with all GPU layers and 36 CPU MoE layers:

./build/bin/llama-cli \
  -m K2-Horizon-MoVA-36B-A4B-Q5_K_M.gguf \
  -ngl 999 -ncmoe 36

Benchmark

Measured locally on an NVIDIA GeForce RTX 3090 (24,103 MiB) and AMD Ryzen 9 9900X.

The benchmark used -p 512 -n 128 -b 512 -ub 512 -r 3 -ngl 999 -ncmoe 36.

| Quant | Prompt processing (512 tokens) | Generation (128 tokens) |

|---|---:|---:|

| Q4_K_M | 832.14 ± 6.21 tok/s | 41.00 ± 0.07 tok/s |

| Q5_K_M | 750.78 ± 13.13 tok/s | 36.85 ± 0.08 tok/s |

| Q6_K | 672.93 ± 7.53 tok/s | 32.68 ± 0.22 tok/s |

The benchmark results and short inference checks were run with the K2Horizon branch and

are included alongside this README in the local build logs.

Source

• Base model: IFM/K2-Horizon-MoVA-36B-A4B
• GGUF conversion source: IFM/K2-Horizon-MoVA-36B-A4B-GGUF
• Required llama.cpp fork: MBZUAI-IFM/llama.cpp, model/K2Horizon