---

license: other

license_name: openmdw-1.1

license_link: LICENSE

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

• gguf
• llama.cpp
• nvfp4
• nemotron
• moe

language:

• en
• es
• fr
• de
• it
• ja

---

NVIDIA Nemotron 3.5 Lightning 30B-A3B - NVFP4 GGUF for llama.cpp

!Nemotron 3.5 Lightning NVFP4: MTP draft-depth sweep (decode tok/s) across four GPUs

A GGUF repack of NVIDIA's Nemotron 3.5 Lightning 30B-A3B (NVFP4)

so it runs on llama.cpp's native Blackwell (sm_120) FP4 tensor-core path. The

4-bit NVFP4 expert weights are copied bit-exact into GGUF; the remaining

tensors (attention, token embedding, Mamba-2 projections, and the MTP head) are

stored at Q8_0. The 4-bit weights are never re-quantized, so the result is

numerically faithful to NVIDIA's checkpoint.

About the model

Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + attention model: 30B total

parameters (~3B active), 52 layers of which only 6 use full attention, 128 routed

experts (6 active per token) plus 1 shared expert, and a native context length of

1,048,576 tokens.

Because only 6 of the 52 layers keep a growing KV cache (the Mamba-2 and MoE layers

carry a fixed-size state), the model holds its **full 1M-token context on a single

24 GB GPU** - something a dense model of this size cannot do.

Files

| File | Size | What it is |

|---|---|---|

| Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf | 18.5 GB | Base model. Runs standalone; full 1M context on a 24 GB card. |

| mtp-Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf | 2.0 GB | Optional MTP head. A speculative-decoding draft, loaded via -md alongside the base. |

Grab the base model for normal use; add the MTP head only if you want speculative

decoding.

Running it (llama.cpp)

Base model, full 1M context on a 24 GB card:

llama-server -m Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -ngl 99 -c 1048576 -fa on --cache-type-k q4_0 --cache-type-v q4_0

With the MTP head for speculative decoding (draft depth 2 is the sweet spot; it uses

more memory, so reduce the context on a 24 GB card):

llama-server -m Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -md mtp-Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf \
    -ngl 99 -c 524288 -fa on --cache-type-k q4_0 --cache-type-v q4_0 \
    --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-ngl 99 \
    --spec-draft-type-k q8_0 --spec-draft-type-v q8_0

The MTP head is a modest, consistent speedup - roughly +7% on fast GPUs and

low-double-digit percent on slower hardware (see the chart above) - at the cost of

some context headroom on 24 GB. It affects throughput only, never output quality:

the draft's tokens are always verified by the base model.

Quality

Evaluated on three benchmarks from the TNG benchmark suite (MMLU-Pro and

terminal-bench 2.0 on distilled subsets, IFBench in full), for the **released

configuration** (base model + external MTP head):

| Benchmark | Score |

|---|---|

| MMLU-Pro | 79.2% |

| IFBench | 70.7% |

| terminal-bench 2.0 | 26.7% |

The repack is numerically faithful to NVIDIA's NVFP4 checkpoint by construction, and

these scores are consistent with the source model within measurement noise. The

benchmarks use distilled subsets with small sample counts, so single-point

differences are noisy - terminal-bench in particular varies substantially run to

run; treat the numbers as indicative rather than precise.

How it was made

Converted with llama.cpp's convert_hf_to_gguf.py: the routed experts, shared

expert, and LM head stay NVFP4 (repacked bit-exact); the token embedding,

attention projections, Mamba-2 in/out projections, and the MTP head are stored at

Q8_0 (near-lossless). No dequantize-and-requantize step, so the vendor's

calibrated 4-bit weights are preserved exactly.

License and attribution

Released under the OpenMDW License Agreement, version 1.1, inherited from

the base model. Model materials Copyright (c) 2026 NVIDIA Corporation & Affiliates.

This is a format repackage (not a retrain) of

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4.

The only changes are quantization / format conversion to GGUF, documented under

How it was made.

Attribution: please credit NVIDIA for the Nemotron 3.5 Lightning model and

its NVFP4 (TensorRT Model Optimizer) export, alongside this repackage.

Trademarks: "NVIDIA" and "Nemotron" are trademarks of NVIDIA Corporation. The

OpenMDW license does not grant trademark rights; these names are used here only to

identify the upstream model. This repackage is **not affiliated with, sponsored by,

or endorsed by** NVIDIA.