---

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- imatrix

- moe

- mamba2

license: other

license_name: nvidia-open-model-license

license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/

---

NVIDIA-Nemotron-3.5-Lightning-30B-A3B — GGUF

GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16,

built with llama.cpp ba360ef. Every claim below is measured; raw logs are in

the metrics repo.

AD- marks an Atomic Dynamic layout: the two expert tensors are assigned explicitly

instead of left to a preset, and the name carries the assignment —

AD-<down_exps>-<up_exps>, collapsed to one name when both match. Everything outside the

experts is Q8_0; it is 7% of the model, so spending bits there is cheaper than reasoning

about it.

!image_2026-08-12_17-34-03

Files

Measured on eval_neutral held-out, KL divergence against the Q8_0 reference.

tok/s is generation on a single RTX 5090, full offload.

| File | down / up | Size | mean KLD | q99 KLD | top-1 | tok/s |

|---|---|---:|---:|---:|---:|---:|

| …-BF16.gguf (split) | bf16 | 65.9 GB | — | — | — | — |

| …-Q8_0.gguf | q8_0 / q8_0 | 35.0 GB | reference | — | — | 290.6 |

| …-AD-Q5_0_XL.gguf | q8_0 / q5_0 | 28.8 GB | 0.00749 | 0.075 | 95.73% | 304.1 |

| …-Q5_K_M.gguf | q5_1 / q5_0 ¹ | 26.6 GB | 0.01781 | 0.159 | 93.22% | 338.7 |

| …-Q4_K_M.gguf | q8_0 / q5_0 ¹ | 25.0 GB | 0.02542 | 0.219 | 91.88% | 348.5 |

| …-AD-IQ4_NL_L.gguf | q5_1 / iq4_nl | 22.4 GB | 0.01345 | 0.123 | 94.16% | 318.1 |

| …-AD-IQ4_NL.gguf | iq4_nl / iq4_nl | 19.7 GB | 0.01728 | 0.157 | 93.49% | 326.5 |

| …-imatrix.gguf | — | 55 MB | the calibration data | | | |

| dflash-…-BF16.gguf | — | 1.2 GB | speculative drafter | | | |

| dspark-…-BF16.gguf | — | 0.7 GB | see below — does not load | | | |

¹ what llama.cpp actually writes after its fallback rules, not what the preset asked for.

Pick AD-IQ4_NL unless you have a reason not to: it matches Q5_K_M on every metric

while being 6.9 GB smaller, and it is the smallest quant on this model that still benefits

from calibration.

Where the AD layouts win

| | size | mean KLD |

|---|---:|---:|

| AD-IQ4_NL | 19.7 GB | 0.01728 |

| Q5_K_M | 26.6 GB | 0.01781 |

Same quality, 6.9 GB less. And against ggml-org's NVFP4 build at effectively identical size:

| | size | mean KLD | top-1 |

|---|---:|---:|---:|

| AD-IQ4_NL_L | 22.4 GB | 0.01345 | 94.16% |

| NVFP4 (ggml-org) | 22.5 GB | 0.06233 | 86.75% |

4.6× lower divergence at the same size. quantize_nvfp4 discards the importance

matrix (GGML_UNUSED(quant_weights) in ggml-quants.c), so an NVFP4 file cannot be

calibrated at all. That is the whole gap. NVFP4 does win on prompt processing — 14460 vs

11809 t/s at pp512, from native Blackwell FP4 kernels — so if you are prefill-bound rather

than quality-bound it remains the better pick.

What we could not deliver: a usable 16 GB quant

Q2_0 (block 64, 2.25 bpw) is the only sub-4.5 bpw type that survives this model's shapes,

and it is calibration-blind — quantize_q2_0 never forwards the imatrix. Measured, it

does not hold up:

| | size | mean KLD | top-1 |

|---|---:|---:|---:|

| AD-Q2_0_L | 15.5 GB | 1.097 | 53.0% |

| AD-Q2_0 | 11.4 GB | 2.403 | 33.5% |

A top-1 agreement of 33% means the model picks a different token two times out of three.

These are published under experimental/ for reproducibility and should not be used for

real work. There is currently no good 16 GB option for this model, from anyone — the

smallest file here that we would stand behind is AD-IQ4_NL at 19.7 GB. Fixing that needs

a block-64 superblock variant of the K-quant family in llama.cpp, which does not exist yet.

Why there is no 2-bit or 3-bit k/i quant

K- and I-quants store weights in superblocks of 256: 256 weights split into 16

sub-blocks of 16, each carrying a 6-bit scale and 6-bit minimum, all normalised against two

fp16 super-scales. That hierarchy is what makes 2- and 3-bit quantization viable — at 2 bits

you cannot afford an fp16 scale every 32 weights, since the scale alone would cost 0.5 bpw.

The price is that a tensor row must divide by 256. This model's expert tensors do not:

ffn_down_exps   ne0 = 1856 = 64 × 29      1856 % 256 = 64
ffn_up_exps     ne0 = 2688 = 128 × 21     2688 % 256 = 128

Together they are 93% of the model. When a row does not divide, llama.cpp's

tensor_type_fallback() substitutes a block-32 type and logs a warning — it does not fail,

and the file keeps the requested name.

!requested vs actual bitrate

| requested | block | becomes | effective bpw |

|---|---:|---|---:|

| IQ1_S … IQ3_S, IQ4_XS | 256 | IQ4_NL | 4.50 |

| Q2_K, Q3_K | 256 | Q4_0 | 4.50 |

| Q4_K | 256 | Q5_0 | 5.50 |

| Q5_K | 256 | Q5_1 | 6.00 |

| Q6_K | 256 | Q8_0 | 8.50 |

This is why published IQ2_XXS, IQ3_XXS, Q2_K and Q3_K_M files for this model all

land around 18–19 GB, and why Q6_K and Q8_0 come out the same size. **Every published

IQ2_XXS for this model is a 4.5 bpw file.**

Types that survive these shapes, and whether they read the imatrix:

| type | bpw | block | imatrix |

|---|---:|---:|---|

| Q1_0 | 1.125 | 128 | no — up_exps only; down_exps errors out |

| Q2_0 | 2.25 | 64 | no |

| MXFP4 | 4.25 | 32 | no |

| Q4_0 | 4.50 | 32 | yes |

| IQ4_NL | 4.50 | 32 | yes |

| NVFP4 | 4.50 | 64 | no |

| Q5_0 | 5.50 | 32 | yes |

| Q5_1 | 6.00 | 32 | yes |

| Q8_0 | 8.50 | 32 | no (does not need it) |

Not a defect in the model — NVIDIA's own NVFP4 uses group 16/64, which these shapes divide

cleanly. It is specific to llama.cpp's k/i family.

To check any model before you quantize it:

python3 -c "
import json,sys; c=json.load(open(sys.argv[1]))
for k in ('hidden_size','intermediate_size','moe_intermediate_size'):
    v=c.get(k)
    if v: print('%-22s %6d  %%256=%3d' % (k,v,v%256))
" config.json

Speculative decoding: all three paths, measured

The model supports three. MTP is inside every file here; DFlash and DSpark are sidecars.

# MTP — no extra file needed
llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf \
  --spec-type draft-mtp -ngl 99 -c 8192

# DFlash — sidecar, auto-discovered by the dflash- prefix
llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf \
  -md dflash-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf \
  --spec-type draft-dflash -ngl 99 -ngld 99 -c 8192

Generation tok/s, single RTX 5090, full offload, identical prompt and seed, 400 tokens:

| File | none | MTP | DFlash | MTP accept | DFlash accept |

|---|---:|---:|---:|---:|---:|

| AD-Q2_0 | 358.4 | 278.1 | 169.0 | 77.9% | 44.6% |

| Q4_K_M | 348.5 | 236.5 | 161.5 | 71.6% | 43.0% |

| Q5_K_M | 338.7 | 214.1 | 159.8 | 65.4% | 44.2% |

| AD-IQ4_NL | 326.5 | 240.8 | 176.8 | 70.3% | 50.4% |

| AD-IQ4_NL_L | 318.1 | 232.2 | 160.8 | 69.5% | 42.3% |

| AD-Q5_0_XL | 304.1 | 226.5 | 160.3 | 72.1% | 46.3% |

| Q8_0 | 290.6 | 189.5 | 143.1 | 61.4% | 40.4% |

On a fast GPU with full offload, speculation costs throughput on every quant — MTP

lands at 65–78% of baseline, DFlash at 47–55%. Draft quality is not the problem: MTP

accepts 61–78% of drafted tokens with a mean accepted length above 3.

The cost is the rollback. This is a hybrid Mamba2 model, so discarding a rejected draft

means restoring SSM state, and the log shows 47.6 MiB copied per speculation step. We

expected larger quants to flip the result, since the target gets more expensive per token

while the checkpoint cost stays fixed. They did not — the ratio is flat across a 3× size

range, so the copy scales with the work too.

Untested here, and where speculation should still pay: partial offload and CPU inference,

where a token costs far more than 47.6 MiB of VRAM-to-VRAM copy. Measure on your hardware

before enabling it.

DSpark does not load. llama.cpp requires a conf_proj.weight of shape

{n_embd + dspark_markov_rank, 1} (src/models/dflash.cpp:94), but NVIDIA's DSpark

checkpoint ships markov_head.markov_w1 / markov_w2 and no confidence projection, so it

fails with check_tensor_dims: tensor 'conf_proj.weight' not found. The converted file is

published anyway in case someone fixes the runtime side.

Converting either sidecar needs --target-model-dir, since neither carries a tokenizer:

python3 convert_hf_to_gguf.py NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash \
  --outtype bf16 --target-model-dir NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --outfile dflash-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf

imatrix

Calibrated on AtomicChat/calib-corpora,

recipe nemotron-3.5-lightning: 4.72M tokens over 3,035 documents — agentic 26.5%,

code 19.1%, reasoning 15.9%, long-context 12.7%, multilingual 12.8%, vocabulary sweep 5.8%,

structured 4.0%, graphics 3.2%.

Rendered through this model's own chat template, so <|im_start|>, <think>,

<tool_call> and <tool_response> appear as the single special tokens the model actually

sees. Run llama-imatrix with --parse-special, or that markup is tokenized as literal

punctuation and ~40% of the corpus calibrates nothing.

!imatrix convergence

| checkpoint | mean cos | median | min | tensors still moving |

|---:|---:|---:|---:|---:|

| 512 | 0.9810 | 0.99924 | 0.4991 | 30 |

| 2000 | 0.9902 | 0.99906 | 0.7664 | 25 |

| 4000 | 0.9957 | 0.99977 | 0.8527 | 16 |

| 6000 | 0.9973 | 0.99989 | 0.8656 | 8 |

| 8512 | 0.99984 | 0.99999 | 0.9963 | 0 |

| 9008 | 0.99991 | 1.00000 | 0.9969 | 0 |

Past 8.5k chunks no tensor changes measurably, so more calibration data of this kind cannot

change the quantizer's decisions. Reproduce with tools/converge.py in the dataset repo.

All 128 experts in all 46 expert tensors receive routing — 0 dead experts at every corpus

size tested, including 823 chunks. Expert coverage is not what a larger corpus buys.

The MTP block (blk.52) is never executed in a normal forward pass, so it collects no

imatrix data at any corpus size. It is pinned to q4_0 in every file:

--tensor-type 'blk\.52\.=q4_0'.

Architecture

52 layers: 23 Mamba2, 23 MoE, 6 attention (indices 5, 12, 19, 26, 33, 42). 128 experts,

6 active plus 1 shared. hidden_size 2688, moe_intermediate_size 1856, squared-ReLU FFN

— experts have up and down only, no gate. GQA 32/2, head_dim 128, rope_theta 10000,

no sliding window, context 262144.

With only 6 attention layers at 2 KV heads the KV cache costs about 6 KB per token, and

Mamba2 state is a constant ~50 MB per sequence regardless of context length. Long context

is unusually cheap here — budget for weights, not for cache.

Running

llama-server -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf -ngl 99 -c 32768

Needs llama.cpp b10326 or newer for nemotron_h_moe. Prompt format:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

Reproducing

python3 convert_hf_to_gguf.py NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --outtype bf16 --outfile Nemotron-BF16.gguf

llama-imatrix -m Nemotron-Q4_0.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special --save-frequency 500

llama-quantize --imatrix imatrix.gguf \
  --tensor-type 'blk\.52\.=q4_0' \
  --tensor-type 'ffn_down_exps=iq4_nl' \
  --tensor-type 'ffn_up_exps=iq4_nl' \
  Nemotron-BF16.gguf Nemotron-AD-IQ4_NL.gguf Q8_0 64

-b 8192 is not cosmetic: at the default batch the imatrix run takes 2h12m instead of

1h22m on the same hardware.