---

license: apache-2.0

base_model: tencent/Hy3

tags:

• gguf
• quantized
• mixed-precision
• prismaquant
• vllm

---

Hy3 295B-A21B — PrismaQuant 2.8-bit GGUF (vLLM-servable)

Tencent Hy3 (the July 2026 production release, Apache 2.0 — not the

preview) — 295B total / 21B active MoE, 80 layers, 192 routed experts top-8 —

quantized by PrismaQuant from the

BF16 source to a 103.7 GB GGUF (2.80 bits/param over quantizable

weights) that serves on a single NVIDIA DGX Spark (128 GB unified memory)

via vLLM + the GGUF plugin.

This supersedes the earlier Hy3-preview-based 2.9-bit artifact: production

base model, Apache 2.0 license, and a quantization menu extended with the

GGUF IQ formats (IQ2_XXS…IQ4_XS, E8-lattice codebooks) alongside the

k-quants — the IQ rungs measured strictly better per byte than the k-rungs

they displace at these bit-rates.

How it was made

• Measured allocation: per-tensor formats chosen by a knapsack solve over

measured per-(tensor, format) quantization error weighted by an empirical

Fisher probe of the BF16 model (32×1024 calibration, streaming — the model

never fits in memory), with per-format calibration factors derived from

end-to-end KL measurements on a smaller proxy model.

• Byte-budget selection: the bit-rate targets a 103.5 GB file (single-Spark

serving with KV-cache headroom) rather than a curve heuristic. The measured

rate-distortion knee sits at ~2.3 bpp — this artifact rides well above it.

• Format mix: routed-expert bulk on IQ2_XS/IQ3_XXS/IQ4_XS; small

attention/shared tensors on Q5_K/Q6_K/Q8_0; embeddings Q4_K; output head

Q6_K; norms/router F32. Full map in allocation/layer_config.json.

• imatrix-weighted quantization (per-column activation second moments from

the same calibration corpus), exhaustive GPU codeword search for IQ formats.

• The MTP layer (model.layers.80) is not included (GGUF serving does not use

speculative decoding here).

Validation — read this

No quality claims are made for this artifact. Models of this scale cannot

be KL-validated against their BF16 teacher on the target hardware. Validation

performed: vLLM loads the artifact, greedy and chat generation are coherent

across smoke prompts (math, prose, technical explanation), and the pipeline's

per-tensor packing is bit-exact against its own emulation and gguf-py

decoding. Community measurements welcome.

Serving (vLLM + GGUF plugin, single DGX Spark)

See serving/: serve.sh launches a vLLM container with the

vllm-gguf-plugin and two

small patches (container_patches.py fixes an upstream embed_tokens

quant-config gap; hy_v3_gguf_adapter.py maps this artifact's

HF-checkpoint-verbatim tensor names onto vLLM's Hy3 loader, including the

stacked 192-expert MoE tensors). Note: all 2-D weight tensors in this

artifact are quantized (no BF16 Linears) — required by the current plugin

load path. llama.cpp does not implement this architecture at time of writing;

serving is vLLM-only.

serving/serve.sh hy3-prismaquant-2.8bpp-00001-of-00003.gguf ./config

Provenance

Quantized with PrismaQuant (streaming direct-from-safetensors GGUF exporter).

Source checkpoint: tencent/Hy3 (BF16). Allocation, Pareto sweep, and

per-format calibration factors are in allocation/. Contact:

robert.tand@icloud.com