satgeze/Hy3-1M-GGUF overview

---

license: apache-2.0

pipeline_tag: text-generation

base_model: tencent/Hy3

tags:

• gguf
• long-context
• yarn
• hunyuan
• hy_v3
• moe
• llama.cpp

---

!Tencent Hy3 1M GGUF

<div align="center">



![Params-2563eb)]()

![Context: 1M-8A2BE2)]()

![First GGUF]()

![NIAH]()

![1M-f9ab00)]()


</div>

<p align="center"><b>Hy3 (Hunyuan) 1M Context GGUF</b>: first GGUF conversion of Tencent's 299B hy_v3, extended to a million tokens. Needle-certified, honestly labeled.</p>

<div style="border: 3px solid #1e8e3e; background: #f0f9f2; border-radius: 12px; padding: 16px 20px; margin: 16px 0;">

<p style="margin:0; font-size: 16px; color: #14532d;"><b>Status, precisely (updated July 9, 2026)</b></p>

<p style="margin: 8px 0 0 0; color: #1e3a29; font-size: 14px;">

Needle-certified <b>10/10 at every rung from 262K to 786K</b> (q8 KV, IQ2_M; Q4_K_M also 10/10 at 524K). At the full <b>1,048,576</b> window: <b>70 percent per-needle across three published seeds</b> (9/10, 6/10, 6/10), so 1M is functional with real edge variance, not perfect. MTP speculative decoding was challenged during upstream review and survived a three-part correctness audit including a behavioral match against Tencent's official vLLM implementation (details below). As of 2026-07-14, <a href="https://github.com/ggml-org/llama.cpp/pull/25395">llama.cpp PR #25395</a> (hy_v3 + MTP) is merged into mainline, so a recent mainline build runs these files natively -- no fork needed. Every claim on this card links to raw evidence in this repo.

</p>

</div>

Tencent's Hy3 (299B MoE, ~17B active, hy_v3 architecture) quantized to GGUF and extended to a 1,048,576-token context with YaRN. As far as we can tell these are the first GGUF quants of Hy3 anywhere: at publish time (roughly 30 hours after Hy3's release) only MLX and MXFP4 quants existed, and mainline llama.cpp did not yet support hy_v3.

Honest status

• Architecture support: native hy_v3 + MTP support is merged into mainline llama.cpp (PR #25395, merged 2026-07-14; the --no-mtp converter gate followed in #25641). Build any recent mainline llama.cpp and these files run natively. The original community fork (charlie12345/ROCmFPX) is credited in the PR; the pre-merge branch build is kept only as a historical note below.
• Quants: Q2_K is a static quant (no importance matrix). IQ2_M and IQ1_M are importance-matrix quants - the imatrix was computed on our own Q2_K running RAM-resident, over a code-and-reasoning calibration set, then used to requantize from the archived f16. This is a bootstrap: a better imatrix could later be computed from a higher-fidelity runner.
• Long context, measured: full NIAH ladder published below (July 9, 2026): 10/10 at 262K, 393K, 524K and 786K; 70 percent per-needle at 1M over three seeds. Multi-hop chain retrieval 4/5 at 524K on both IQ2_M and Q4_K_M.
• MTP: working, audited. We ported hy_v3 MTP inference to llama.cpp ourselves. When upstream review questioned the acceptance behavior we ran a full correctness program: wiring verified line-by-line against the vLLM reference, quantization ruled out (f16 head behaves the same), and a behavioral cross-check against Tencent's official vLLM MTP on identical greedy prompts: vLLM 46.7 percent ungated acceptance vs our 46.9 percent, a dead match. Conclusion: the single-depth head is inherently confidence-gated; run it with --spec-draft-p-min 0.75 (88 percent acceptance, up to +37 percent decode on CUDA). See the MTP section below.
• Chat template: Hy3's stock template uses .format() calls that llama.cpp's Jinja engine rejects. chat_template_llamacpp.jinja in this repo is the fixed version; pass it with --chat-template-file.

What the gate actually produced (real, unedited)

Each quant was asked: "Name three colors and write a one-line haiku about the sea." Verbatim outputs, nothing cleaned up:

IQ1_M (62 GB, the roughest quant):

> blue waves that whisper low,

> green depths that hold the silence of the sea.

IQ2_M (90 GB):

> Three colors: blue, teal, and silver.

> waves drink the moonlight

> saltwind hums through silent blue-

> the sea forgets names

Q2_K (101 GB):

> Blue, green, silver.

> The sea breathes in mist,

> silver light on rolling waves-

> quiet, endless blue.

My honest read

This is better than "it didn't collapse into repetition." A 1.7-bit quant of a 299B model (IQ1_M) still produces grammatical, on-theme lines; the two larger quants produce properly structured 5-7-5 haiku and correctly separate the color list from the poem. That is a genuinely good sign for how much of the model survives aggressive quantization here.

Two honest caveats, so you judge for yourself:

• This is a single short prompt. It proves coherence and basic instruction-following only. For reasoning and long-context retrieval, see the certification ladder above.
• IQ1_M is visibly the weakest: it dropped the explicit color list and gave two lines instead of a haiku. Usable, but the imatrix IQ2_M is the one I would actually run if the memory fits.

If your experience differs, the raw files are right here - try it and form your own view.

Needle-in-a-haystack certification (July 9, 2026)

<img src="niah_heatmap.png" width="720"/>

Run on rented H200s (a 299B with full attention needs 86 GB of KV at 524K even at q8; no consumer card can host these rungs). Note the shape of the 1M column: the misses cluster at 25 to 45 percent depth, the classic lost-in-the-middle zone, while the edges stay solid. 10 needles per rung, depths 5 to 95 percent, temperature 0, seeded haystacks, q8 KV, MTP on with p-min 0.75. Raw records: results_niah_ladder.jsonl in this repo.

| Rung | IQ2_M | Q4_K_M |

|---|---|---|

| 262,144 | 10/10 | |

| 393,216 | 10/10 | |

| 524,288 | 10/10 | 10/10 |

| 786,432 | 10/10 | |

| 1,048,576 | 9/10, 6/10, 6/10 (three seeds) | |

Reading the 1M row honestly: 21 of 30 needles over three independent seeds, misses at scattered depths, identical scores independently reproduced at q4 KV (9/10, same seed as the 9/10 run). Through 786K this model is needle-perfect; at the very edge of the baked window it retrieves about 70 percent per-needle. We publish that as "functional at 1M", not "certified at 1M", and both labels are earned.

Multi-hop tier (chained lookups, each answer keys the next, thinking ON): chains 4/5 at 524K on IQ2_M and 4/5 on Q4_K_M, 2/5 at 262K (seed variance). A harder multi-step variable-tracking tier is held back until our scorer for it is validated; unverified numbers do not get published here.

MTP: challenged in review, audited, confirmed correct

During upstream review an NVIDIA engineer questioned whether low ungated draft acceptance meant the MTP wiring was wrong. We ran the full program instead of arguing:

<img src="mtp_audit.png" width="760"/>

| Test | Method | Result |

|---|---|---|

| Code parity | Our graph vs vLLM reference implementation, all four flagged axes (concat order, norm offset, pre/post-norm hidden, RMSNorm form) | Faithful on all four |

| Quantization ruled out | f16 full-precision head, greedy, ungated | 46.9 percent (IQ2_M: 41.0), so quant is not the cause |

| Behavioral cross-check | Tencent's official vLLM MTP path, bf16, greedy, identical prompts, vLLM's own metrics | 46.7 percent ungated: a dead match with our 46.9 |

Conclusion: Hy3's single-depth MTP head genuinely proposes low-confidence drafts when ungated, in every implementation including the official one. It is designed to be confidence-gated. Practical guidance, measured:

| Condition (greedy) | Acceptance | Speed (2xH200, IQ2_M) |

|---|---|---|

| No speculation | n/a | 72.2 tok/s |

| MTP ungated (p-min 0) | 41.0 percent | 73.6 tok/s (break-even) |

| MTP gated (p-min 0.75) | 88.2 percent | 81.5 tok/s (+13 percent) |

On Apple silicon the gate is what keeps MTP from being a net loss; on CUDA ungated is break-even and the gate turns it into a gain. Earlier wording on this card described ungated MTP as a universal net slowdown; that was Metal-specific and is corrected here. Raw logs: results_mtp_audit/ in this repo.

Files

| File | Size | Where | Notes |

|---|---|---|---|

| hy3-1M-MTP-IQ2_M.gguf | 100 GB | HF + ModelScope | Recommended. The certified quant (table above), MTP baked |

| hy3-1M-MTP-Q2_K.gguf | 111 GB | HF + ModelScope | Static quant + MTP |

| hy3-1M-MTP-IQ3_XXS.gguf | 117 GB | HF + ModelScope | imatrix quant + MTP |

| hy3-1M-MTP-Q3_K_M.gguf | 145 GB | HF + ModelScope | Sweet spot for 2x96GB rigs, MTP baked |

| hy3-1M-MTP-Q4_K_M.gguf | 183 GB | HF + ModelScope (split parts) | Quality pick for 192GB+ rigs, 10/10 at 524K |

| hy3-1M-MTP-Q5_K_M.gguf | 199 GB | HF + ModelScope (split parts) | Near-lossless tier |

| hy3-1M-MTP-Q6_K.gguf | ~245 GB | HF + ModelScope (split parts) | Highest quant of the ladder, near-lossless |

| hy3-1M-IQ1_M.gguf | 62 GB | ModelScope only | Smallest; fits a 128 GB Mac. no_think only; NOT needle-certified |

| hy3-1M-IQ2_M.gguf / hy3-1M-Q2_K.gguf | 90 / 101 GB | HF + ModelScope | MTP-less originals, kept for history |

| chat_template_llamacpp.jinja | small | HF + ModelScope | Fixed template for llama.cpp |

On ModelScope, files over the size cap arrive as -00001-of-0000N split parts; point llama.cpp at the first part and it finds the rest.

Field testing notes (real sessions, July 7)

Measured on a MacBook Pro M3 Max 128 GB, Metal, fork build, IQ1_M fully GPU-resident:

• Speed: 24-25 tok/s generation, ~41 tok/s prefill (llama-server timings, short prompts). A 299B MoE at chat speed on a laptop.
• no_think mode: coherent and usable, but noticeably chatty: filler sentences, over-explaining, occasional odd asides. Fine for casual chat and drafting; does not feel like a frontier chat tune.
• Thinking mode (reasoning_effort: high): collapses at IQ1_M. On a bare "hello" it produced hundreds of words of circular reasoning ("The user wrote hello... we should respond... but wait, the user wrote hello...") without ever converging to an answer. This is the 1.7-bit tax: short answers survive, reasoning chains do not. Do not use IQ1_M for coding or anything that needs thinking.
• Because of that, IQ1_M lives on ModelScope only; HF carries the quants we can recommend. IQ2_M field results will be added after the same test battery.
• Known cosmetic issue on the ROCmFPX fork build: the <|hy_eos:opensource|> stop token can leak as visible text at the end of responses. We root-caused it: the GGUF tokenizer metadata is correct (the token is properly CONTROL-typed); the leak is in that fork's chat parser, which renders special tokens for structural parsing and passes unknown trailing ones through. Mainline llama.cpp strips specials before parsing, so it does not have this issue (the leak was specific to that fork chat parser). Workarounds on the fork: add the token as an explicit stop string, or apply the parser patches we are submitting to that repo. Generation always stops correctly; the text is just noise.

MTP speculative decoding: measured guidance (July 8)

The -MTP- quants carry Hy3's draft head (protected at q8_0). Measured on these exact files with mainline llama.cpp (PR #25395, merged):

| Platform | MTP effect | Setting |

|---|---|---|

| CUDA (RTX 5090) | +26 to 37% decode | --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 |

| Apple silicon (M3 Max, IQ2_M) | neutral: 23.27 vs 23.21 tok/s, 88.7% acceptance | flags optional, no penalty with p-min 0.75 |

--spec-draft-p-min 0.75 is required either way: the MTP head is trained single-depth, and without the confidence gate speculation becomes a net slowdown (38.6% acceptance measured). Full data and discussion: Apple-silicon datapoint.

Speed expectations, honestly: quoted tok/s are fresh-context ceilings; in an ongoing conversation expect roughly 17 to 19 tok/s on the M3 Max at IQ2_M as context deepens. Both numbers are normal physics.

How to control thinking with llama.cpp: pass --chat-template-kwargs '{"reasoning_effort": "high"}' (or no_think / low) at server launch.

How to actually use a 1M-context model

Habits that measurably help, from our RULER, hop and adherence testing across this fleet:

1. Re-state standing instructions near the end of long prompts; recency beats depth.
2. One big reference dump beats a long accumulated conversation. Fresh session per task.
3. After any compaction or summarization, repeat your active rules yourself.
4. Prefill at 500K+ takes real time on any hardware; stage your questions accordingly.
5. Know your quant: IQ2_M holds retrieval to 786K; IQ1_M is a no_think chat quant, not a work quant.

Running with mainline llama.cpp

hy_v3 + MTP support is merged into mainline llama.cpp (ggml-org/llama.cpp#25395, merged 2026-07-14; AI-implemented with full disclosure, validated on these exact GGUFs). Build a recent mainline llama.cpp and it runs these files natively -- no fork or custom branch needed.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Apple Silicon (Metal):
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

# NVIDIA (CUDA). If you build inside a conda env and hit
# "undefined reference to cuMemGetInfo", point at the real driver lib:
cmake -B build -DGGML_CUDA=ON \
  -DCUDA_cuda_driver_LIBRARY=/usr/lib/x86_64-linux-gnu/libcuda.so.1 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

MTP speculative decoding note for this model: launch with --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75. The p-min 0.75 matters: Hy3's MTP head is trained single-depth, and without the confidence gate speculation becomes a net slowdown (measured 38.6 percent acceptance vs 97.3 percent with it).

Historical note (pre-merge): before #25395 landed, these quants were first built on the community fork charlie12345/ROCmFPX (which required stubbing four empty web-UI files: tools/server/public/{bundle.css,bundle.js,loading.html,index.html}), then on the hy3-mtp branch. Both are credited in the PR. You no longer need either -- mainline runs these files.

Run it (mainline llama.cpp, 2026-07-14 or later)

llama-server -m hy3-1M-IQ2_M.gguf -c 262144 -np 1 --jinja \
  --chat-template-file chat_template_llamacpp.jinja

Start at 262K and raise -c toward 1048576 per available memory. On Apple Silicon the IQ1_M runs; on 192 GB+ the IQ2_M is the quality pick.

Credits

Base model: Tencent (Apache-2.0). hy_v3 llama.cpp implementation: community fork (charlie12345). GGUF conversion, 1M YaRN extension, imatrix bootstrap, and gating: SatGeze. Method and tooling: github.com/satindergrewal/aviary-1m.