---

license: apache-2.0

base_model:

• trohrbaugh/Hy3-heretic

base_model_relation: quantized

language:

• en
• zh

pipeline_tag: text-generation

library_name: gguf

tags:

• gguf
• hy3
• hunyuan
• moe
• mixture-of-experts
• heretic
• abliterated
• uncensored
• imatrix
• speculative-decoding
• mtp

---

Hy3-Heretic-295B-A21B — IQ2_XXS (attention-protected) with MTP restored

An imatrix GGUF quantization of trohrbaugh/Hy3-heretic, the

decensored/abliterated variant of tencent/Hy3 (295B total / 21B active MoE).

Two things make this build unique:

1. It fits on a single 96 GB card with 64K context, fully resident in VRAM. The next-smallest

llama.cpp-compatible Hy3-heretic quant is 100.01 GiB — larger than the card before you add any KV cache.

1. It has a working MTP head, so speculative decoding actually runs. No other Hy3-heretic release has this

at any size, because they all inherit a source checkpoint whose MTP layer was lost. This one was grafted back.

---

Quick start

llama-server \
  --model Hy3-Heretic-MTP-IQ2_XXS-attnQ4K.gguf \
  -ngl 999 --ctx-size 65536 \
  -fa on -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 1024 \
  --jinja --chat-template-file hy3-chat-template.jinja \
  --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 \
  --temp 0.9 --top-p 1.0

Sampling defaults are per the Tencent model card (temp 0.9, top-p 1.0). Drop the --spec-* flags to run without

speculative decoding.

---

Requirements

Needs a llama.cpp build from 2026-07-13 or later. hy_v3 support (with MTP) was added in

PR #25395, merged 2026-07-13. Earlier builds will fail with

unknown model architecture: 'hy_v3'. There is no ik_llama.cpp support for this architecture.

Use the fixed chat template, not the embedded one. The template inherited from the Tencent original renders the

EOS token as literal text after assistant turns, which leaks <|hy_eos:opensource|> into content and breaks

streaming tool calls. Pass a corrected jinja file with --chat-template-file.

---

Files

| file | size |

|---|---|

| Hy3-Heretic-MTP-IQ2_XXS-attnQ4K.gguf | 75.78 GiB (81,364,399,584 bytes) |

Single file — 2.18 BPW, 1298 tensors, block_count 81, nextn_predict_layers 1.

hf download prometheusAIR/Hy3-heretic-MTP-GGUF \
  Hy3-Heretic-MTP-IQ2_XXS-attnQ4K.gguf --local-dir .

---

Fit on a single RTX PRO 6000 Blackwell (96 GB)

Measured usable VRAM on that card is ~95.6 GiB.

| | weights | + 64K q8_0 KV | + compute | total | fits? |

|---|---|---|---|---|---|

| mradermacher Q2_K | 100.01 GiB | — | — | over before KV | ✗ |

| this build | 75.78 GiB | 10.6 GiB | ~2.5 GiB | ~88.9 GiB | ✓ (~6.7 GiB spare) |

Measured resident footprint in practice: 89,854 MiB of 97,887.

---

Quantization recipe

Built with an imatrix (bartowski/Hy3-GGUF/Hy3-imatrix.gguf,

876 entries / 812 chunks). Bits are deliberately spent on the dense path rather than the experts:

| tensor | type | why |

|---|---|---|

| attn_q, attn_k, attn_v, attn_output | Q4_K | attention is dense — every parameter is read for every token in all 80 layers, so its error compounds with depth |

| token_embd | Q8_0 | it is a gather, not a matmul: higher precision costs ~nothing in speed |

| output | Q6_K | vocab-sized matmul each token |

| ffn_*_exps (routed experts) | IQ2_XXS | sparse — only 8 of 192 fire per token, so error is diluted and routing partially self-corrects |

| all of blk.80 (MTP layer) | Q4_K | see the imatrix note below |

Census: 488 F32 / 467 IQ2_XXS / 331 Q4_K / 10 Q2_K / 1 Q6_K / 1 Q8_0.

No iq1_s, iq2_s or iq3_s tensors — see the sm_120 warning below.

Why the dense path is protected — measured, not assumed

An earlier build of this same model used the stock llama-quantize IQ2_XXS recipe, which leaves attn_q and

attn_output at ~2.06 bpw. Boosting only the dense path to the table above (+2.02 GiB, 2.09 → 2.15 BPW) produced

a measurable behavioural change on a multi-step reasoning task with a 900-token budget:

| build | result | length |

|---|---|---|

| stock recipe (2-bit attention) | truncated, no answer emitted | 2613 chars |

| this recipe (Q4_K attention) | completed, correct answer | 2178 chars |

Cost: 9.1% throughput. Worth it.

---

MTP restoration

What was done: layer 80 was extracted from tencent/Hy3 (593 tensors, 3.753B params) and grafted into the

abliterated weight set, taking the index from 46,545 to 47,138 tensors — exactly Tencent's own count. Trunk layers

0–79 are untouched abliterated weights; this was verified by sha256 against a non-MTP build of the same source

(token_embd, output, and blk.0/40/79 attention and expert tensors are byte-for-byte identical).

*The head is unmatched*** — Tencent's original MTP head paired with an abliterated trunk. This costs almost nothing:

| prompt | acceptance here (unmatched) | reference: base Hy3, matched head |

|---|---|---|

| short reasoning | 91.8% | 87.1% |

| code | 94.6% | 93.5% |

| 4K depth | 90.0% | 89.1% |

Speculative decoding's acceptance test preserves the target model's sampling distribution, so a mismatched head

affects speed, not quality.

Running with MTP

--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75

p-min 0.75 is required: Hy3's single-depth head is confidence-gated by design, and ungated acceptance is far

lower in every implementation.

---

Measured performance

RTX PRO 6000 Blackwell 96 GB, fully VRAM-resident, 64K context, -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024:

| prompt | with MTP | without MTP | gain |

|---|---|---|---|

| short reasoning | 107.39 tok/s | 96.70 | +11.1% |

| code | 121.11 tok/s | 96.89 | +25.0% |

| 4K depth | 88.27 tok/s | 86.77 | +1.7% |

| prompt processing @4K | ~2420 tok/s | | |

Cold load ~16 s from NVMe.

Correctness: needle-in-a-haystack exact at 1K / 4K / 16K / 32K / 56K tokens and at depths 0.25–0.9 (7/7,

largest verified prompt 57,925 tokens). Streaming tool calls return finish_reason=tool_calls with correct

arguments. Strict-JSON instruction following clean. Generated code passes executable doctests.

---

Limitations — please read

• The abliteration's effectiveness is UNVERIFIED here. A 6-prompt battery of legitimate requests that aligned

models tend to over-refuse scored 0/6 refusals on this build — but also 0/6 on the non-abliterated base model,

so the probe cannot discriminate. Whether the Heretic edit survived 2.18bpw quantization was not established.

Evaluate it yourself against your own use case.

• It is roughly 50–70% more verbose than non-abliterated Hy3. On a task the base model finished in 945

characters, this build used ~1550–2180. Budget max_tokens generously; a truncated answer here is often a

correct answer that ran out of room. Part of that verbosity was quantization (fixed by the attention boost above)

and part is the abliteration itself.

• Structured reasoning is measurably weaker than base Hy3. On a bin-packing battery graded for conservation,

capacity and optimality, abliterated builds score 2/5 where non-abliterated builds score 3/5 — consistent across

two different quantization profiles, so this is an abliteration cost rather than a quantization one.

• This is a ~2.18 bpw quant of a 295B model. If it fits in your VRAM budget, a larger quant will be better.
• Inherits any biases and failure modes of tencent/Hy3 and of the Heretic abliteration process.

---

Notes for anyone reproducing this

Three things cost real time and are not obvious:

1. No imatrix covers blk.80. An imatrix is captured from ordinary forward passes and the MTP head never fires

during those, so every imatrix for this architecture has that hole. Since

ggml_quantize_requires_imatrix() is true for IQ2_XXS / IQ2_XS / IQ1_S, quantizing blk.80 to IQ2_XXS

aborts the entire run at the last tensor, after roughly an hour of work. Force the layer to a type that needs

no imatrix:

```

--tensor-type 'blk\.80\.=q4_K'

```

It must come first — overrides are std::regex_search with first-match-wins. (bartowski hit the same wall

and solved it with Q4_0 for all of blk.80; that is the otherwise-inexplicable "11 Q4_0" in his IQ2_XXS census.)

1. --dry-run does not validate imatrix coverage. It computes target types and reports a size without running

the imatrix check, so a clean dry-run is not a green light. Grep its output for blk.80 target types.

1. Do not copy shards wholesale when grafting. Layer 80 lives in 5 of Tencent's 99 shards, but 3 of those also

contain 183 tensors belonging to layers 0–79. Copying those files in would silently revert those tensors to

non-abliterated Tencent weights, partially un-abliterating the model with no error anywhere. Re-serialize only

model.layers.80.*.

⚠️ Blackwell / sm_120 users

On the hardware this was built and tested on (compute capability 12.0 — RTX PRO 6000 Blackwell and RTX 5070 Ti),

llama.cpp's CUDA matmul kernels for iq1_s, iq2_s and iq3_s produce garbage: verified with

test-backend-ops, errors of 0.52–1.40 against a 5e-4 tolerance, on both MUL_MAT and MUL_MAT_ID. The failure

is silent — no load error, just wrong output.

This matters because a quant's name does not tell you which ggml types are inside it. For example

bartowski/Hy3-GGUF IQ2_XS contains 18 iq2_s tensors, and IQ2_M is built on iq2_s plus iq3_s.

This build deliberately contains none of those three types. Verified on one host and one llama.cpp build, so treat

it as a strong warning rather than a universal claim — but if you are on Blackwell and a low-bit i-quant is

producing nonsense, this is likely why.

---

Credits

• tencent/Hy3 — base model, and the source of the grafted MTP layer
• trohrbaugh/Hy3-heretic — the abliterated weights
• Heretic by p-e-w — the abliteration method
• bartowski — the importance matrix, and the Q4_0-on-blk.80 trick
• llama.cpp — and PR #25395 for hy_v3 + MTP support