---

base_model: tencent/Hy4-preview

base_model_relation: quantized

quantized_by: qtum

license: apache-2.0

license_link: https://huggingface.co/tencent/Hy4-preview/blob/main/LICENSE

language:

• en
• zh

pipeline_tag: text-generation

tags:

• gguf
• llama.cpp
• hunyuan
• hy4
• moe
• quantized
• imatrix

---

Hy4-preview GGUF

GGUF quantizations of tencent/Hy4-preview, made with llama.cpp.

Chinese version: README_zh.md

780B total parameters, 78 layers. DeepSeek-style MLA attention (q-LoRA rank 2048,

kv-LoRA rank 512) with a gated output path and learnable attention sinks, wrapped in

iHC hyper-connections on every attention and MLP block. 256 routed MoE experts with

top-8 routing plus one shared expert. DSA sparse attention: 21 of the 78 layers own a

full indexer, the rest share the preceding full layer's top-k. Vocabulary 120,832.

Quantized from the official BF16 weights. Every tier is imatrix-calibrated and ships

as 39 shards.

Quantizations

| Tier | Size | Shards | BPW | PPL (wikitext-2) | Mean KLD | Same top-1 |

|---|---|---|---|---|---|---|

| master (BF16, not in this repo) | 1435.7 GiB | 39 | 16.02 | 2.9361 ± 0.0253 | — | — |

| IQ4_XS | 388.3 GiB | 39 | 4.33 | 3.0789 ± 0.0157 | 0.0793 ± 0.0008 | 90.50 % |

| Q2_K | 269.6 GiB | 39 | 3.01 | 3.8898 ± 0.0217 | 0.3451 ± 0.0030 | 80.35 % |

| IQ2_XS | 218.9 GiB | 39 | 2.44 | 4.9648 ± 0.0294 | 0.5988 ± 0.0044 | 73.44 % |

The master row is not a file in this repo. It is listed so the numbers above have a

reference point — the 1.4 TiB BF16 master was actually loaded and measured on the same

machine, so every tier is compared against a real baseline rather than an estimate.

Pick IQ4_XS unless you cannot fit it. It keeps the same top-1 token as the BF16

master on 90 % of positions and its KL divergence is 4.4× lower than the next tier down.

Q2_K is the balanced choice and the best option for CPU-only inference. IQ2_XS exists

for when the footprint is hard-capped.

What the imatrix is worth

Every tier here is calibrated with an importance matrix. To quantify what that is worth,

Q2_K was built twice — once with the imatrix and once without — at **byte-identical

size** (276,118 MiB both times), so the only variable is the calibration:

| | PPL | Mean KLD | Same top-1 | RMS Δp |

|---|---|---|---|---|

| Q2_K without imatrix | 4.2988 ± 0.0238 | 0.4543 ± 0.0034 | 76.98 % | 24.82 % |

| Q2_K as shipped | 3.8898 ± 0.0217 | 0.3451 ± 0.0030 | 80.35 % | 21.36 % |

| improvement | −9.5 % | −24.0 % | +3.4 pts | −13.9 % |

Measured against this repo's own size-quality curve, that 24 % KL reduction is worth

+0.257 BPW — i.e. reaching the same quality without an imatrix would need a file

8.5 % larger (about 23 GiB more on the Q2_K tier). The imatrix itself costs nothing

in the shipped weights.

Where the gain lands, by KL percentile:

| Percentile | without | with | gain |

|---|---|---|---|

| 10 % | 0.00155 | 0.00058 | −62.8 % |

| median | 0.16232 | 0.09597 | −40.9 % |

| 90 % | 1.2245 | 0.9229 | −24.6 % |

| 99 % | 3.7199 | 3.3953 | −8.7 % |

| 99.9 % | 6.5440 | 6.4235 | −1.8 % |

The improvement is concentrated in the body of the distribution, not the tail: an imatrix

makes the typical token markedly closer to the original model, while the rare

catastrophic divergences are a property of low-bit quantization that calibration cannot

undo. This is exactly what per-channel importance weighting is expected to do.

Calibration corpus: 476 KiB of real-world text, 220 chunks at n_ctx=512, run over the

full BF16 master (6.7 h on 8× H100).

What is protected

The routed experts are 96.5 % of this model's weights, so everything else can be kept at

high precision for almost no size cost. It is kept:

| Tensors | Type | Reason |

|---|---|---|

| ffn_gate_inp / exp_probs_b | F32 | MoE router; compressing it routes tokens to the wrong experts |

| ffn_*_shexp | Q8_0 | the shared expert runs on every token |

| hc_attn_ / hc_ffn_ / output_hc_* | F32 | iHC hyper-connection streams, every layer |

| indexer.k_norm / indexer.proj | F32 | DSA top-k selection; a wrong pick discards the whole attention span |

| indexer.attn_k / indexer.attn_q_b | Q8_0 | DSA indexer projections |

| attn_kv_a_mqa / attn_k_b / attn_v_b / attn_q_a | Q8_0 | MLA low-rank paths are already narrow; compressing them damages the KV cache |

| attn_gate / attn_sinks | Q8_0 / F32 | gated-MLA output path and learnable sinks |

| ffn_gate / ffn_up / ffn_down | Q6_K | the single dense MLP layer before the MoE stack |

| token_embd / output | Q6_K | a global type would otherwise squeeze these hard |

Usage

# Point at the first shard; llama.cpp finds the rest on its own.
llama-cli -m Hy4-preview-IQ4_XS-00001-of-00039.gguf

Do not pass -ngl manually. llama.cpp fits layers to free VRAM by itself, and an

explicit -ngl — including 0 — aborts that fitting and can fail to allocate compute

buffers even when the weights would have fit.

Every tier ships as 39 shards. Download all 39 into one directory — you only ever name

-00001-of-00039 on the command line.

Requires a llama.cpp build with hy_v4 support (merged upstream 2026-09-04, PR #28127).

Earlier builds will not load these files.

About the numbers

wikitext-2 test, n_ctx=512, 655 chunks (~335 K tokens), every tier through the exact

same command on 8× H100 80GB. KL divergence and top-1 agreement are computed against

logits saved from the BF16 master over 200 chunks.

These numbers are only comparable within this table. Do not compare them against PPL

figures published by other repos — different corpora and chunk counts make absolute values

meaningless across setups.

Two notes on why the error bars here are small enough to be useful: the PPL runs use 655

chunks rather than the handful common in quantization write-ups, and KL divergence is

reported alongside PPL because perplexity is a mean statistic that hides where a quantized

model actually breaks. The percentile table above is only readable because both were done.

License

Apache-2.0, inherited from tencent/Hy4-preview —

see LICENSE for terms.

Quantized by qtum.