---

language:

- en

- es

- fr

- de

- ja

- it

license: other

license_name: nvidia-open-model-license

license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

pipeline_tag: text-generation

tags:

- gguf

- nemotron

- moe

- local

- quantized

- padded

---

Nemotron 3.5 Lightning 30B A3B — padded IQ3_XXS (fork-only GGUF)

A 14.33 GiB GGUF of NVIDIA-Nemotron-3.5-Lightning-30B-A3B that serves the full

262,144-token context on a single 16 GB GPU — with **genuine IQ3_XXS expert

tensors**, which no stock build of this model can produce.

⚠️ This file requires a patched llama.cpp. Two separate reasons (see Building):

the expert tensors are zero-padded, and this model needs upstream Dflash support

that only landed on 2026-08-11.

Why this exists

Nemotron-3.5-Lightning's MoE expert tensors are 1856 / 2688 / 3712 wide. None is

divisible by 256, so llama-quantize silently falls back to block-32 types for every

block-256 (K/IQ) quant — the model's file size stops responding to the requested

bit-width.

This is not theoretical. Reading the tensor census straight out of the published

unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF/…-UD-IQ3_XXS.gguf (19.76 GB):

| tensor group | actual type |

|---|---|

| 46 trunk expert tensors | IQ4_NL (block-32, ~4.5 bpw) |

| blk.52 MTP experts | Q5_0 |

| everything else | Q8_0 ×95, Q6_K ×30, F32 ×243, BF16 ×1 |

| IQ3_XXS tensors | 0 |

A file named IQ3_XXS that contains no IQ3_XXS at all. Every published sub-4-bit GGUF

of this family has the same shape.

The fix: zero-pad the reduction axes so block-256 quants actually apply.

| tensor | original | padded |

|---|---|---|

| ffn_up_exps reduction axis | 2688 | 2816 |

| ffn_down_exps (expert FF width) | 1856 | 2048 |

| shared-expert FF width | 3712 | 3840 |

This is exact, not approximate: the MoE activation is relu², and relu²(0) = 0,

so the padded rows contribute exactly zero. Zero is exactly representable in every ggml

block format, so the padding stays inert after quantization.

Result: genuine IQ3_XXS experts, and the whole 262k context fits a 16 GB card.

Measured (RTX 5080 16 GB, sm_120, CUDA 13.0, WSL2)

| metric | value |

|---|---|

| File | 15,387,569,504 bytes (14.33 GiB) |

| VRAM @ 262,144 ctx, q4_0 KV, after load | 14,784 MiB |

| VRAM @ full 250k-token prompt | 14,878 MiB (1,100 MiB free) |

| Prefill @ 250k | 3,819 t/s |

| Decode @ 250k depth | 78.8 t/s |

| Generation smoke test | 3/3 pass |

Quality benchmarks (MBPP / HumanEval / agentic) have not been run on this build yet —

see Status below. Nothing here claims a quality result.

sha256

0ba17723bce5f3e7faaf1941c6a3c393249f379c790fcb55d72e221095821aa8  Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf

Building llama.cpp (required)

Stock llama.cpp cannot load this file. You need current master (for Dflash) plus

the padding patch included here.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# Must include PR #26905 "Dflash support for nemotron-3.5" (merged 2026-08-11).
# Any master checkout after that date is fine.
git apply /path/to/nemotron-expert-padding.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=120        # sm_120 = RTX 5080; use your own GPU's capability
cmake --build build --config Release -j

The patch is 3 files / 21 inserted lines: read the padded width from a GGUF key, size

ffn_up_exps from it (in both the trunk and MTP paths), ggml_pad the MoE input to

match, and size the per-expert output views from the down projection.

> Toolchain caveat: NVCC 13.2 miscompiles the i-quant CUDA kernels

> (iq1_s/iq2_s/iq3_s fail test-backend-ops test -o MUL_MAT). This build was made

> with CUDA 13.0. If you are on 13.2+, rebuild against 13.0 before trusting IQ output.

Serving

llama-server \
  -m Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf \
  --host 127.0.0.1 --port 8189 --alias lightning \
  -ngl 99 -np 1 --flash-attn on \
  -c 262144 -ctk q4_0 -ctv q4_0 \
  --jinja --temp 0.6 --top-p 0.95

serve-lightning.sh wraps this. Sampling, per NVIDIA's card for the family: **0.6 / 0.95

for tool calling (the served default above), 1.0 / 1.0 for reasoning** — set

per-request. Thinking is on by default; disable per-request with

"chat_template_kwargs": {"enable_thinking": false}.

The MTP head

Lightning ships a multi-token-prediction head — blk.52, declared via block_count = 53

and nextn_predict_layers = 1. It is a full extra MoE layer (its own 128 experts).

It is retained in this file, with its experts at Q5_0. Two things to know:

1. It costs disk, not VRAM. llama.cpp sets mtp_flags = !load_mtp ? TENSOR_SKIP : 0,

and load_mtp is false unless you pass --spec-type draft-mtp. On a normal load the

server logs model has unused tensor blk.52.… -- ignoring for all 16 of them, and

they are never allocated. The 14,878 MiB measured above is with the MTP head in the

file.

1. Its experts cannot be quantized below ~4 bpw, by anyone. llama-imatrix only

observes tensors on the normal forward path, and the MTP head isn't on it — so no

importance data exists for blk.52, and llama-quantize correctly refuses

("Missing importance matrix … in a very low-bit quantization … bailing out"). Q5_0 is

a block-32 type that needs none. This is the same approach unsloth shipped.

**Speculative decoding via --spec-type draft-mtp is available but has not been tested

here.** If you try it, results are welcome in the discussions.

> Do not try to shrink this by pruning layer 52: --prune-layers 52 updates

> block_count but not the per-layer array KVs, and the result fails to load with

> key nemotron_h_moe.feed_forward_length has wrong array length; expected 52, got 53.

Status / what's still open

• imatrix — waiting on unsloth. This build uses bartowski's Lightning imatrix

(padded to the padded geometry; included as imatrix-padded-lightning.gguf). Unsloth's

own quant metadata references

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF/imatrix_unsloth.gguf

(unsloth_calibration_NVIDIA-Nemotron-3.5-Lightning-30B-A3B.txt, 80 chunks, 185

entries), but that file is not present in their repo — they published one for

Nemotron-3-Nano but not for Lightning. We have asked. If it appears, this model will be

rebuilt with it and the two compared directly.

• Benchmarks not yet run on this build (MBPP / HumanEval / agentic).
• MTP speculative decoding untested.

Files

| file | purpose |

|---|---|

| Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf | the model |

| nemotron-expert-padding.patch | required llama.cpp patch (3 files, 21 lines) |

| imatrix-padded-lightning.gguf | bartowski's Lightning imatrix, padded to match |

| pad_gguf.py | pads a BF16 master (the technique) |

| pad_imatrix.py | pads an imatrix to the padded geometry |

| verify_pad.py | asserts the padded regions are exactly zero |

| serve-lightning.sh | the serve command above |

Reproduction

# 1. pad a BF16 master (e.g. unsloth's 2-shard BF16 GGUF)
python pad_gguf.py --variant b --in <master-*.gguf> --out lightning-padded-bf16.gguf
# 2. pad the imatrix to match
python pad_imatrix.py --variant b --in <imatrix.gguf> --out imatrix-padded-lightning.gguf
# 3. quantize with the patched build.
#    blk.52 rules MUST come first -- the first matching --tensor-type rule wins.
llama-quantize --imatrix imatrix-padded-lightning.gguf \
  --tensor-type "blk\.52\.ffn_up_exps=q5_0" \
  --tensor-type "blk\.52\.ffn_down_exps=q5_0" \
  --tensor-type ffn_up_exps=iq3_xxs \
  --tensor-type ffn_down_exps=iq3_xxs \
  lightning-padded-bf16.gguf out.gguf Q4_K_M 14

Provenance & licence

• Base: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, architecture

nemotron_h_moe (Mamba2 + MoE hybrid, 128 experts, 52 trunk layers + 1 MTP layer,

native 262,144 context).

• Licence: NVIDIA Open Model License — see the base model card. Verify compliance for

commercial use before redistributing.

• The padding technique was developed 2026-08-10 for Nemotron-3-Nano-30B-A3B; see

Nemotron-3-Nano-30B-A3B-pirola-IQ3_XXS-GGUF

and the 12 GB variant

Nemotron-3-Nano-30B-A3B-pirola-IQ2_XXS-XS-GGUF.

• Upstream discussion on making this unnecessary:

ggml-org/llama.cpp#27037.