---

license: other

license_name: tencent-hy-community

license_link: LICENSE

base_model: tencent/Hy3-preview

tags:

- gguf

- quantization

- mixed-precision

- prismaquant

- vllm

- moe

---

Hy3 (295B-A21B) — PrismaQuant measured-allocation GGUF, 2.89 bpp (107 GB)

Tencent Hy3-preview (295B total / 21B active MoE, 80 layers, 192 routed

experts top-8 + 1 shared expert, 256K-context family) quantized from the

BF16 source to a 107.0 GB GGUF that serves on a **single NVIDIA DGX

Spark (GB10, 128 GB unified memory)** via vLLM's GGUF path.

This is not a hand recipe. Every 2-D tensor's format was chosen by

AURA — PrismaQuant's production-faithful KL–Fisher allocator — which

selects each tensor's hardware format by measured cost under a byte

budget rather than by heuristics:

• Per-Linear empirical Fisher sensitivity (H_trace) from a calibration

backward pass over the BF16 source (streaming; the model never fits in

memory whole), 32×1024 calibration tokens, 1024 activation rows. The

allocation objective is the AURA KL–Fisher expansion:

Δloss ≈ ½ · H_trace · quantization-MSE per (tensor, format).

• Per-(tensor, format) quantization error measured for every tensor against

the GGUF k-quant ladder (Q2_K 2.625 / Q3_K 3.4375 / Q4_K 4.5 /

Q5_K 5.5 / Q6_K 6.5625 / Q8_0 8.5 bpw), with activation-aware (imatrix)

scale selection — per-column importance from the same calibration

activations, applied identically at cost measurement and export.

• A multi-choice knapsack over 718 decision units (stacked MoE expert

tensors are single units — one format per (layer, projection), all 192

experts uniform) selecting the measured Pareto knee at 2.888 bpp

under a byte budget targeting ~105–107 GB on disk.

What the allocator chose

| format | units | params |

|---|---|---|

| Q2_K | 159 | ~246B (routed-expert bulk) |

| Q3_K | 153 | ~17B |

| Q4_K | 115 | ~28B |

| Q5_K | 151 | ~1.2B |

| Q6_K | 21 | ~0.4B |

| Q8_0 | 119 | ~1.3B |

The expert mass is pressed to 2.6–3.4 bpw while several hundred small,

sensitive Linears (attention projections, shared-expert matrices) won

higher rungs on measured Δloss — an allocation shape no uniform preset

produces. Embeddings ship Q4_K, the LM head Q6_K, norms and the

routing-critical expert_bias at F32, router gates F16. The MTP layer

(model.layers.80) is not included (no speculative decoding).

Full allocation provenance is in this repo (allocation/) and baked into

the GGUF KV metadata (prismaquant.*: git commit, assignment SHA-256,

imatrix SHA-256, per-tensor format map).

⚠️ No quality claims

No perplexity/KL/benchmark claims are made for this artifact. Rigorous

quality measurement of a 295B model requires serving its BF16 reference,

which needs hardware far beyond a single Spark — that is the point of this

artifact. What has been verified: the model loads, holds coherent

long-form generation and correct code at temperature 0, and decodes at

~18 tok/s on the GB10. Expect low-bit tail artifacts (rare single-token

glitches) — this is a 2.9 bpp quantization of a 295B model, not magic.

Serving (vLLM + gguf plugin; llama.cpp is NOT supported)

llama.cpp has no hy_v3 architecture (upstream request:

ggml-org/llama.cpp#22477), so this artifact targets vLLM with the

out-of-tree vllm-gguf-plugin.

Two integration pieces ship in serving/:

1. hy_v3_adapter.py — a plugin weights adapter (tensor names in this

GGUF are HF-checkpoint-verbatim; stacked expert tensors load through

vLLM's 3-D full-load path).

1. container_patches.py — a one-line vLLM fix: upstream hy_v3.py

constructs embed_tokens without quant_config, so quantized

embeddings cannot load without it (pending upstream).

See serving/serve.sh for the full recipe (vLLM ≥ 0.23/0.24 with the

HYV3 architecture, plugin installed with its CUDA kernels, config +

tokenizer sidecars from config/). Notes: all 2-D Linears in this

artifact are quantized (required — this arch does not thread module

prefixes, so the plugin cannot exempt unquantized Linears), KV-cache fp8

recommended, and on a 128 GB host use eager mode with modest context.

Provenance & method

Built with PrismaQuant / AURA (Robert Tand, robert.tand@icloud.com) —

AURA: Production-Faithful KL–Fisher Allocation, mixed-precision

quantization that chooses each tensor's hardware format by measured cost

rather than heuristics. See the other PrismaQuant/AURA artifacts under

rdtand for the method applied across

model families and format menus (NVFP4/FP8 compressed-tensors, GGUF). Source checkpoint:

tencent/Hy3-preview (BF16). Quantized bytes are bit-exact against the

ggml/gguf-py reference layouts. Base model license applies (Tencent Hy

Community License — see LICENSE; note its territorial exclusions).