---

license: apache-2.0

base_model: tencent/Hy3

base_model_relation: quantized

pipeline_tag: text-generation

quantized_by: vcruz305

tags:

- gguf

- imatrix

- moe

- hy3

- tencent

- conversational

---

Hy3-GGUF

imatrix GGUF quantizations of tencent/Hy3 — 295B total / 21B active MoE (192 experts, top-8), 80 layers + 1 MTP/NextN layer (3.8B), 256K context.

Quantized day-zero from the BF16 release and smoke-tested on real hardware before upload (NVIDIA DGX Spark, GB10). Performance numbers below are measured, not estimated.

⚠️ Requires llama.cpp PR #25364 (unmerged)

The hy_v3 architecture is not yet in llama.cpp master. Until PR #25364 merges, build from the PR branch:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/25364/head:hy3-port && git checkout hy3-port
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j"$(nproc)"

These quants were produced at PR head a4da4b5cfdc4e5fa9def068e216a6e5154f22848.

Quants

All quants use an importance matrix (Hy3.imatrix, included) computed on a ~63KB diverse coding/reasoning/chat calibration corpus. Files are sharded at ~48GB for HF's 50GB limit — download the whole folder and point llama.cpp at the -00001-of-* shard; the rest load automatically.

| Quant | Size | ~BPW | Fits | Notes |

|---|---|---|---|---|

| Q8_0 | 318 GB | 8.6 | server-class | near-lossless |

| Q5_K_M | 212 GB | 5.8 | 2× 128GB-class | |

| Q4_K_M | 181 GB | 4.9 | 2× 128GB-class | recommended dual-node; verified over RPC |

| IQ4_XS | 159 GB | 4.3 | 2× 128GB-class | |

| Q3_K_M | 143 GB | 3.9 | 2× 128GB-class | |

| IQ3_XXS | 117 GB | 3.2 | single 128GB-class (borderline) | MTP block @ q8_0; best quality-per-GB single-box |

| Q2_K | 109 GB | 3.0 | single 128GB-class | |

| IQ2_M | 100 GB | 2.7 | single 128GB-class | MTP block @ q8_0 |

| IQ1_M | 92 GB | 2.5 | single 96GB-class (tight) | mixed-precision recipe from Tencent's AngelSlim release (attn q8_0, ffn_down_exps iq3_xxs, early layers iq2_xxs) + this repo's imatrix; MTP eh_proj @ q8_0; smallest tier |

On low-bit IQ tiers the MTP/NextN layer (blk.80.*) is kept at q8_0 (--tensor-type) — very-low-bit quantization of that block is not possible without imatrix coverage, and this preserves it intact for future speculative decoding support.

Measured performance (DGX Spark, GB10, 273GB/s unified memory)

Single node, fully GPU-resident, -fa on:

| Quant | Gen tok/s | + MTP spec decode | Prompt tok/s |

|---|---|---|---|

| IQ2_M | 18.0–18.5 | 22.8 (+27%) | 36–50 |

| Q2_K | 18.1 | untested | 35.5 |

| IQ3_XXS | 17.1 | untested | 32.4 |

| IQ1_M | 14.4–14.7 | 23.1 (+58%, 88–92% acceptance) | 33–43 |

Notable: IQ1_M's MTP head is quantized at q4_K/q5_K (per the AngelSlim recipe) rather than q8_0, and draft acceptance is statistically identical to the q8_0 head on IQ2_M — the MTP block tolerates 4-bit quantization with no measurable acceptance loss. Since speculative decoding is verify-always, draft-head precision can only affect speed, never output quality.

MTP numbers measured at temp 0 with --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75 (90% draft acceptance). Higher sampling temperatures reduce acceptance and land between the two columns.

Dual node (Q4_K_M, 181GB layer-split across 2× GB10 over 200GbE via llama.cpp RPC): 14.0 tok/s gen / 21.9 tok/s prompt. RPC layer-split adds capacity for bigger quants, not speed — expect single-node-or-slower decode rates.

All smoke-tested tiers (IQ1_M, IQ2_M, Q2_K, IQ3_XXS single-node; Q4_K_M dual-node) produced coherent output (code generation + chat), with the Hy3 chat template engaging correctly via --jinja. The remaining tiers (Q8_0, Q5_K_M, IQ4_XS, Q3_K_M) were produced by the same verified pipeline but not individually load-tested — report any issues in the community tab.

Running

# chat/completion (--jinja is required — the Hy3 template is not natively supported)
./build/bin/llama-completion \
  -m Hy3-Q2_K/Hy3-Q2_K-00001-of-00003.gguf \
  -ngl 99 -fa on -c 8192 --jinja \
  -p "Write a Python function that returns the median of a list." -n 256

# server
./build/bin/llama-server \
  -m Hy3-Q2_K/Hy3-Q2_K-00001-of-00003.gguf \
  -ngl 99 -fa on -c 8192 --jinja --host 0.0.0.0 --port 8080

Tips:

• First load of a 100GB+ quant can take several minutes — don't kill it early.
• Constrained on memory? Offload MoE expert tensors to CPU with --n-cpu-moe N.
• --jinja matters: without it the chat template aborts on current llama.cpp.

MTP / speculative decoding status

Verified working (2026-07-07): PR #25395 adds Hy3 MTP speculative decoding, and these GGUFs work with it as-is — the MTP tensors bundled in every quant (q8_0-preserved on low-bit tiers) are used directly as the draft-mtp target. Measured on IQ2_M: 18.0 → 22.8 tok/s (+27%), 90% draft acceptance.

./build/bin/llama-server -m Hy3-IQ2_M/Hy3-IQ2_M-00001-of-00003.gguf \
  -ngl 99 -fa on -c 32768 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75 \
  --parallel 1

Notes:

• --spec-draft-p-min 0.75 is required for a speedup — the MTP head is trained single-depth, and the default p_min makes speculation a net loss (per the PR author's measurements, confirmed here).
• --parallel 1 is required for draft-mtp; n_max 2 and 3 measure within ~1% of each other (n=2 slightly ahead at 90% acceptance).
• --spec-type exists on llama-server and llama-cli only, not llama-completion.

⚠️ If you downloaded before 2026-07-08: arch string fix

PR #25395 renamed the architecture string from hy-v3 (the earlier PR #25364) to hy_v3. All first-shards in this repo were re-uploaded with the fix on 2026-07-07, so fresh downloads just work. If you hold older files and see unknown model architecture: 'hy-v3', either re-download the first shard of your quant, or patch in place (the string lives only in shard 00001's header; byte-for-byte same length):

# python patch_arch.py <your-first-shard.gguf>  — swaps hy-v3 -> hy_v3 in the header
import mmap, sys
with open(sys.argv[1], "r+b") as f:
    mm = mmap.mmap(f.fileno(), 64 * 1024 * 1024)  # metadata lives well within 64MB
    i = mm.find(b"hy-v3")
    while i != -1:
        mm[i:i+5] = b"hy_v3"; i = mm.find(b"hy-v3", i + 1)
    mm.flush()

Provenance

• Source: tencent/Hy3 (BF16, 597.6GB, 99 shards)
• llama.cpp: PR #25364 @ a4da4b5cfdc4e5fa9def068e216a6e5154f22848
• imatrix computed on a Q8_0 intermediate (standard practice for models whose BF16 GGUF exceeds node RAM)
• Quantized and validated on a 2-node NVIDIA DGX Spark cluster

Quantized by vcruz305. Please report issues in the community tab.