---

license: apache-2.0

base_model:

• soyaakinohara/K2-Horizon-32B-heretic
• IFM/K2-Horizon-32B

language:

• en
• multilingual

library_name: gguf

pipeline_tag: text-generation

tags:

• gguf
• llama.cpp
• k2_horizon
• 32b
• uncensored
• refusal-removed
• heretic
• quantized
• bf16
• q8_0
• q6_k
• q4_k_m
• q2_k

---

K2-Horizon-32B Heretic GGUF

GGUF quants of a refusal-attenuated derivative of the

IFM/K2-Horizon-32B model.

The pipeline is: Heretic refusal attenuation (trial 124: 3/100 refusals,

KL divergence 0.056) → LoRA merge → GGUF conversion → quantization.

This release includes all five precision levels.

---

Files

K2-Horizon-32B-heretic-BF16.gguf
K2-Horizon-32B-heretic-Q8_0.gguf
K2-Horizon-32B-heretic-Q6_K.gguf
K2-Horizon-32B-heretic-Q4_K_M.gguf
K2-Horizon-32B-heretic-Q2_K.gguf

| File | Size | Note |

|---|---|---|

| K2-Horizon-32B-heretic-BF16.gguf | ~69.6 GiB | Conversion source. Highest precision |

| K2-Horizon-32B-heretic-Q8_0.gguf | ~37.0 GiB | No imatrix. Needs 40+ GiB VRAM to fully offload |

| K2-Horizon-32B-heretic-Q6_K.gguf | ~28.5 GiB | No imatrix. Fits 2x16 GiB VRAM fully offloaded |

| K2-Horizon-32B-heretic-Q4_K_M.gguf | ~21.1 GiB | Recommended. imatrix-calibrated. Fits 2x16 GiB VRAM |

| K2-Horizon-32B-heretic-Q2_K.gguf | ~13.3 GiB | Smallest. imatrix-calibrated |

| Property | Value |

|---|---|

| Architecture | K2Horizon (custom) |

| Parameters | 34.8B dense |

| Format | GGUF |

| Context | 524,288 tokens max (reduce on small VRAM, see below) |

| License | Apache-2.0, inherited from the base model |

---

llama.cpp usage

A K2-compatible llama.cpp build is required (upstream does not support this

architecture). Use MBZUAI-IFM/llama.cpp

at branch model/K2Horizon.

Server example (2x16 GiB VRAM):

llama-server -m K2-Horizon-32B-heretic-Q4_K_M.gguf \
  --split-mode tensor --tensor-split 1,1 \
  --host 0.0.0.0 --port 8080 -ctk q4_0 -ctv q4_0 -c 8192

Note: -c 32768 needs about 17 GiB extra for KV cache and will OOM on

2x16 GiB cards. Keep -c 8192 with q4 KV cache on this class of GPU.

---

日本語

概要

IFM/K2-Horizon-32B

をベースに、Heretic で無検閲化したモデルの GGUF です。

trial 124(拒否 3/100、KL 0.056)をマージしています。

BF16・Q8_0・Q6_K・Q4_K_M・Q2_K の5段階入りです。

使い方

llama.cpp は K2 対応フォークが必要です(本家は未対応)。

llama-server -m K2-Horizon-32B-heretic-Q4_K_M.gguf \
  --split-mode tensor --tensor-split 1,1 \
  --host 0.0.0.0 --port 8080 -ctk q4_0 -ctv q4_0 -c 8192

VRAM 16GB×2 での実用は Q4_K_M が目安です。

ctx を 32768 まで上げると KV で溢れるので、

長い ctx が要る場合は -c を抑えるか KV 量子化を使ってください。