---

license: other

license_name: nvidia-open-model-license

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

tags:

- gguf

- magicquant

- hybrid-quantization

- nemotron

---

NVIDIA-Nemotron-3.5-Lightning-30B-A3B — MagicQuant hybrid GGUFs

Per-group hybrid quantizations of

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

(hybrid-SSM MoE, ~30B total / A3B active), produced by MagicQuant's **measured

evolutionary search**: tensors are grouped by role, candidate per-group scheme

assignments are actually rendered and perplexity-measured (not predicted),

and winners are selected per size tier. imatrix-calibrated, KL-scored

(kl_weight 0.1), 15 measured candidates over 3 rounds.

Pins: MagicQuant 14e9d62, Foundry 2f99202. Search ledger and the full

pre-registered comparison record live with the run.

Files

| file | size | PPL (wikitext-2, ctx 512, full corpus) | vs BF16 (6.83) |

|---|---|---|---|

| ...-Q4_K_M.gguf | 17.60 GiB | 6.8948 ± 0.046 | +0.95% |

| ...-Q5_K_M.gguf | 20.60 GiB | 6.8579 | +0.41% |

| ...-Q6_K.gguf | 24.13 GiB | 6.7979 | −0.47% (within noise of baseline) |

All PPLs from one CPU llama.cpp instrument (ctx 512, full wiki.test.raw,

same baseline). Q4 was re-measured standalone with error bars; Q5/Q6 values

are the search's internal measurements on the identical instrument (the

standalone Q4 re-measurement reproduced the search-internal value to 4

decimals).

Tier filenames follow effective-bpw bands, not a single uniform quant type —

each file mixes schemes per group (e.g. the Q4 winner spends its budget as

D:Q4_K_M E:Q8_0 H:BF16 K:BF16 O:BF16 Q:Q8_0 S:Q8_0 U:IQ4_NL X:MXFP4_MOE).

How the Q4 winner was validated

As a controlled experiment, an alternative per-tensor budget-knapsack

allocation (MagicQuant v2, which predicts damage from a distortion table

instead of measuring candidates) was built at exactly the Q4 winner's size,

under identical pinned code, and measured on the same instrument:

> At matched 17.6 GiB, the measured per-group search beat the predicted

> per-tensor allocation: +0.95% vs +3.47% quality loss — a +2.50% PPL gap,

> far outside the measurement error.

The losing artifact is not published (dominated at identical size); the

numbers above are from the pre-registered comparison record.

Notes

• Chat template embedded and verified present in every file before upload.
• SSM/state tensors are held at F32/F16 per llama.cpp requirements (the

never-quantize and f32-required-operand classes); BF16-designated groups

are written as F16 on disk (llama.cpp BF16 compute-graph limitation).

• Requires a llama.cpp recent enough for this architecture and for

MXFP4_MOE/IQ4_NL tensor types (mid-2026 or newer).

• Built and measured on an AMD Strix Halo (Ryzen AI MAX+ 395, gfx1151) host.