---

license: apache-2.0

tags:

- hunyuan

- hy3

- moe

- text-generation

- ik_llama

- GGUF

base_model: tencent/HunYuan-3

quantized_by: noonr48

---

Tencent Hy3 (HunYuan-3) — IQ4_NL GGUF

295B parameter Mixture-of-Experts model quantized to IQ4_NL (4.5 bits per weight), 158 GB.

⚠️ Requires a Custom Fork of ik_llama

This model will not load on standard llama.cpp or standard ik_llama.cpp. It requires a fork with hy_v3 architecture support:

→ https://github.com/noonr48/ik_llama-hy3 (branch: hy3-support)**

The fork adds:

• hy_v3 architecture enum, hparams, and tensor loading
• Graph builder dispatch (shares build_glm4_moe())
• Custom chat template handler with thinking token parsing
• MTP speculative decoding support (built-in NextN layer)
• EOS token leak fix

Standard llama.cpp does not recognize the hy_v3 architecture and will refuse to load this model.

Model Details

| Property | Value |

|----------|-------|

| Architecture | hy_v3 (MoE with shared expert, sigmoid gating) |

| Total parameters | 295B (192 experts × ~1.5B + shared expert) |

| Active parameters per token | ~10B (8 of 192 experts + 1 shared expert) |

| Layers | 81 (80 transformer + 1 MTP/NextN) |

| Embedding dimension | 4096 |

| Attention heads | 64 (8 KV heads, GQA) |

| Training context length | 262,144 (256K) |

| RoPE base frequency | 11,158,840 |

| Vocabulary | 120,832 tokens |

| Quantization | IQ4_NL (4.5 bpw) |

| File size | 158 GB |

How This Model Was Made

1. Source model: Tencent HunYuan-3 (Apache 2.0 license), downloaded from HuggingFace in bf16 format
2. Conversion: Streaming HF → GGUF conversion using the custom fork's convert_hf_to_gguf.py (modified to detect and handle hy_v3 architecture)
3. Calibration: Importance matrix (imatrix) generated from a calibration dataset containing:

- Open-source code samples (various languages, Apache/BSD/MIT licensed)

- OEIS integer sequence data (public domain mathematical sequences)

- General text passages

1. Quantization: IQ4_NL quantization applied using the imatrix for optimal weight selection (717 imatrix entries across 128 chunks)
2. Verification: Model loaded, inference tested, output quality verified

What's Tested and Working

| Feature | Status | Notes |

|---------|--------|-------|

| Model loading (IQ4_NL, 158 GB) | ✅ | Loads on multi-GPU (10 GPUs, graph split) |

| Inference (token generation) | ✅ | Clean output, correct token stream |

| Chat template (custom Jinja2) | ✅ | Thinking tokens parsed, reasoning extracted |

| EOS token handling | ✅ | Fixed leak where EOS text appeared in output |

| Reasoning separation | ✅ | --reasoning-format deepseek splits thinking from answer |

| MTP speculative decoding | ✅ | 69–80% draft acceptance rate, ~54% speedup on large prompts |

| Graph split across multiple GPUs | ✅ | 11 splits across 10 GPUs verified |

| 32K context (KV cache on GPU) | ✅ | ~35–46 tok/s generation |

| --flash-attn on | ✅ | Required for this architecture |

| CUDA: sm_86 (RTX 3090) | ✅ | |

| CUDA: sm_120 (RTX 5060 Ti) | ✅ | |

Known Limitations

• EOS token ID override required: The GGUF metadata contains eos_token_id=3 (incorrect). You must pass --override-kv tokenizer.ggml.eos_token_id=int:120025 at runtime.
• 200K context is slow with CPU KV offload: With --no-kv-offload at 200K context, large prompts (3K+ tokens) process at ~5 tok/s due to PCIe bandwidth. Use 32K all-GPU KV for interactive use (~40 tok/s).
• MTP n_max=2 not recommended: Causes OOM on 24 GB GPUs with large prompts. n_max=1 is faster (higher acceptance rate) and more memory-efficient.
• --ubatch-size 512 can OOM with MTP: Use --ubatch-size 256 when MTP is enabled.

Quick Start

Build the fork

git clone -b hy3-support https://github.com/noonr48/ik_llama-hy3.git
cd ik_llama-hy3
cmake -B build -DGGML_CUDA=ON -DLLAMA_BUILD_TESTS=OFF
cmake --build build --target llama-server -j$(nproc)

Serve (32K context, recommended for interactive use)

./build/bin/llama-server \
  --model Hy3-IQ4_NL-v7.gguf \
  --host 0.0.0.0 --port 9999 \
  --n-gpu-layers 999 \
  -sm graph \
  --override-kv tokenizer.ggml.eos_token_id=int:120025 \
  --ctx-size 32768 --batch-size 512 --ubatch-size 256 \
  --flash-attn on --cache-type-k f16 --cache-type-v f16 \
  --jinja --chat-template-file models/templates/Hy3.jinja \
  --reasoning-format deepseek --reasoning on \
  --spec-type mtp:n_max=1,p_min=0.0

Serve (200K context, for batch / long-context tasks)

./build/bin/llama-server \
  --model Hy3-IQ4_NL-v7.gguf \
  --host 0.0.0.0 --port 9999 \
  --n-gpu-layers 999 \
  --no-kv-offload \
  --override-kv tokenizer.ggml.eos_token_id=int:120025 \
  --ctx-size 200000 --batch-size 512 --ubatch-size 256 \
  --flash-attn on --cache-type-k f16 --cache-type-v f16 \
  --jinja --chat-template-file models/templates/Hy3.jinja \
  --reasoning-format deepseek --reasoning on \
  --spec-type mtp:n_max=1,p_min=0.0

MTP Speculative Decoding

Hy3 includes a built-in Multi-Token Prediction layer (block 80, NextN architecture). When enabled, the model predicts the next token using a lightweight draft pass, then verifies it with the full model. This reduces the number of full forward passes by ~30-50%.

Enable with: --spec-type mtp:n_max=1,p_min=0.0

Measured speedup: ~54% faster on large prompts (3K+ tokens), ~33% faster on short prompts.

Hardware Requirements

• Minimum: ~160 GB total VRAM (e.g., 7× RTX 3090, or equivalent)
• Tested on: 10-GPU mixed fleet (1× RTX 5090, 3× RTX 3090, 6× RTX 5060 Ti), ~199 GB total VRAM
• CPU RAM: 64 GB minimum for model loading (more if using CPU KV offload)

Chat Template

A custom Jinja2 chat template is required for correct thinking token handling. The template handles:

• <think:opensource></think:opensource> thinking markers
• <｜hy_eos:opensource｜> EOS token
• System/user/assistant message formatting

The template file (models/templates/Hy3.jinja) is included in the GitHub fork repository.

Credits

• Original model: Tencent HunYuan-3 (Apache 2.0)
• Inference engine: ik_llama.cpp by Ikawrakow
• hy_v3 fork: noonr48/ik_llama-hy3
• MTP infrastructure: Originally built for GLM-DSA / Qwen3.5-MoE in ik_llama.cpp

License

Apache 2.0 (inherited from the original Tencent HunYuan-3 model)