---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

pipeline_tag: text-generation

tags:

• gguf
• nemotron
• moe
• imatrix
• mamba

---

NVIDIA Nemotron 3.5 Lightning 30B-A3B - GGUF

Community GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.

These files contain the standard 52-layer inference model. The trailing MTP prediction head was removed during BF16-to-GGUF conversion with --no-mtp; it was not deleted from an already-quantized file. Every published quant was produced from that metadata-consistent no-MTP BF16 GGUF and load-and-generation tested before upload.

Available files

| File / quant | Status | Size (GB) | Size (GiB) | Spark prompt tok/s | Spark generation tok/s | Recommended GPU examples | Tensor fallback / notes |

| --- | --- | ---: | ---: | ---: | ---: | --- | --- |

| Nemotron-3.5-Lightning-30B-A3B-MIXED-Q2_0-Q4_0-2.47BPW.gguf | Uploaded | 9.76 | 9.09 | 140.5 | 113.1 | 12 GB: RTX 3060 12GB, RTX 4070 / 4070 Super, RTX 5070. 8 GB via partial offload. | Experimental mixed Q2_0/Q4_0; no fallback |

| Nemotron-3.5-Lightning-30B-A3B-MIXED-Q2_0-Q4_0-2.97BPW.gguf | Uploaded | 11.74 | 10.93 | 139.6 | 108.4 | 16 GB: RTX 4060 Ti 16GB, RTX 4070 Ti Super, RTX 5070 Ti, RTX 5080, RX 7800 XT | Experimental mixed Q2_0/Q4_0; no fallback |

| Nemotron-3.5-Lightning-30B-A3B-MXFP4_MOE.gguf | Uploaded | 17.98 | 16.75 | 94.4 | 73.7 | 24 GB: RTX 3090, RTX 4090, RTX A5000, RX 7900 XTX | Native MoE-oriented FP4; no fallback |

| Nemotron-3.5-Lightning-30B-A3B-IQ4_NL.gguf | Uploaded | 17.98 | 16.74 | 124.1 | 90.7 | 24 GB: RTX 3090, RTX 4090, RTX A5000, RX 7900 XTX | Strong size/speed option; output promoted to Q8_0 |

| Nemotron-3.5-Lightning-30B-A3B-Q4_0.gguf | Uploaded | 18.06 | 16.82 | 104.8 | 92.6 | 24 GB: RTX 3090, RTX 4090, RTX A5000, RX 7900 XTX | Recommended compatibility default; output promoted to Q8_0 |

| Nemotron-3.5-Lightning-30B-A3B-Q5_0.gguf | Uploaded | 21.96 | 20.45 | 96.6 | 81.7 | 24 GB with limited context; 32 GB preferred, e.g. RTX 5090 or RTX 5000 Ada | Higher-fidelity tier; output promoted to Q8_0 |

| Nemotron-3.5-Lightning-30B-A3B-Q8_0.gguf | Uploaded | 33.59 | 31.28 | 85.9 | 58.9 | 40-48 GB+: A100 40GB, A40 48GB, RTX A6000 48GB, RTX 6000 Ada 48GB | Near-lossless reference; no fallback |

Sizes are decimal GB and binary GiB. GPU examples are practical full-offload targets, not guarantees: runtime memory also depends on context length, KV-cache type, backend, compute buffers, and driver overhead. The TPS figures were measured on the DGX Spark GB10 described below and should not be treated as expected speed on the listed consumer GPUs.

GPU selection guide

For the most headroom, choose a file at least 1-2 GiB below available VRAM before allocating the KV cache. Very long contexts can require substantially more memory than the model file alone.

• 8 GB: partially offload the 2.47 BPW file to the GPU and keep the remaining layers in system RAM.
• 12 GB: 2.47 BPW with a modest context; 2.97 BPW is possible only with very little headroom and carefully chosen KV-cache settings.
• 16 GB: 2.97 BPW is the preferred low-bit option with useful context headroom.
• 24 GB: MXFP4_MOE, IQ4_NL, or Q4_0; Q5_0 can fit but leaves less room for context.
• 32 GB: Q5_0 comfortably; Q8_0 remains too tight for reliable full offload once runtime buffers are included.
• 40-48 GB+: Q8_0 with practical context headroom.

DGX Spark benchmark

The table reports one standardized 128-token run per file on an NVIDIA DGX Spark GB10 with full GPU offload. All runs used the same prompt and these options:

-n 128 -ngl 999 --temp 0 --seed 42 --no-conversation --single-turn --simple-io

The compatible llama.cpp build identified itself as b1-70dfba5. These figures are a local throughput reference, not a formal quality benchmark; prompt length, context, sampling, cooling, and llama.cpp revisions can change results.

Quadro RTX 6000 benchmark and Hermes Agent check

All six sub-24-GiB files were also downloaded and tested on a 24 GB Quadro RTX 6000 (Turing, compute capability 7.5). A Q2_0-capable llama.cpp CUDA build at commit 38406d5 was compiled specifically for SM 7.5. Every file loaded successfully with -ngl 999, including Q5_0 at an 8K server context with Q8_0 KV cache.

Throughput used three repetitions of llama-bench -p 512 -n 128 -ngl 999 -fa on -ctk q8_0 -ctv q8_0. Values are mean tokens per second; these workstation results are separate from the DGX Spark figures above.

| Quant | Size (GiB) | RTX 6000 prompt tok/s | RTX 6000 generation tok/s | Simulated agentic battery | Native Hermes evaluable tests |

| --- | ---: | ---: | ---: | ---: | ---: |

| Mixed Q2_0/Q4_0 2.47 BPW | 9.09 | 2438.3 | 137.9 | 4/6 | 2/3 |

| Mixed Q2_0/Q4_0 2.97 BPW | 10.93 | 2427.6 | 137.0 | 4/6 | 1/3 |

| MXFP4_MOE | 16.75 | 2310.3 | 106.0 | 6/6 | 3/3 |

| IQ4_NL | 16.74 | 2377.1 | 126.7 | 5/6 | 2/3 |

| Q4_0 | 16.82 | 2325.5 | 121.6 | 5/6 | 2/3 |

| Q5_0 | 20.45 | 2165.5 | 115.5 | 5/6 | 3/3 |

The agentic checks used hermes-agentic-bench, temperature 0 for its six-test simulated battery, and the real Hermes CLI for tool restraint, missing-file recovery, and todo-first planning. The native web-search test is excluded from the score because this machine had no Firecrawl credential, so every quant encountered the same unavailable-tool condition. The destructive native file test remained disabled. MXFP4_MOE and Q5_0 were repeated through the native battery and reproduced 3/3 on the evaluable tests.

For this card, MXFP4_MOE is the recommended overall agentic quant: it was the only file to pass all six deterministic simulated scenarios and it matched Q5_0 on the repeated native checks while using about 3.7 GiB less VRAM. The experimental 2.47 BPW file is the throughput and memory-efficiency winner, but its 4/6 simulated result included an unsafe ambiguous-delete call and it should not be treated as equivalent in capability to the 4-5 bit tiers.

Low-bit method and why there is no honest IQ1 file

This model's main MoE expert matrices use dimensions 1856 and 2688. Many llama.cpp IQ and K formats require a dimension divisible by 256. For this architecture, requesting those formats causes 134 of 417 source tensors - including most expert weights - to fall back to roughly 4-bit storage. The resulting IQ1/IQ2/IQ3 labels are therefore misleading because their files converge near the same approximately 17 GB payload after MTP removal.

Stock Q2_0 is dimension-compatible, but direct BF16-to-Q2_0 conversion showed severe output collapse. Source review found that the stock encoder ignored the importance matrix and, with its max-absolute scale, never selected one of the four available codes for ordinary BF16 blocks.

The two experimental low-bit files use a quantizer-only, importance-aware scale optimizer that searches the complete {-1, 0, +1, +2} * d Q2_0 codebook. The resulting GGUF blocks use the standard Q2_0 on-disk and inference representation; no custom inference kernel is required, but a current llama.cpp build with Q2_0 support is required. The reproducibility patch is included as q2_0_weighted_optimizer.patch.

Both low-bit files keep attention, SSM, and shared-expert projections at Q4_0, the token embedding at Q5_0, and the output tensor at Q8_0. The 2.47 BPW file stores the routed expert up/down projections as optimized Q2_0. The 2.97 BPW file additionally protects the 11 highest-impact late expert-down projections (blocks 29, 31, 34, 36, 38, 40, 43, 45, 47, 49, and 51) with Q4_0. These are mixed-precision recipes, not stock or pure 2-bit quants, so the filenames report measured effective BPW.

There is still no honest 1-bit release: the nominal IQ1 route falls back across most expert tensors and produces a roughly 4.7 BPW file. Publishing it as "IQ1" would be misleading.

The seven published formats either have no fallback or only promote explicitly documented tensors. Their filenames describe the actual bulk representation rather than merely the requested quantizer.

Importance matrix and verification

• Source: full NVIDIA BF16 checkpoint.
• MTP handling: removed at conversion with --no-mtp.
• Importance matrix: 400 sequential 512-token chunks from the WikiText-2 portion of the calibration corpus.
• Imatrix generation: full BF16 model resident on a DGX Spark GB10.
• Verification: fresh model load, full GPU offload, and text generation for every uploaded file.

Held-out low-bit quality check

Perplexity was measured on eight 512-token chunks from the appended source-code tail of the calibration file. The imatrix consumed only the earlier WikiText-2 portion, so this code sample was held out from quant calibration. Lower is better.

| Model | Effective BPW | Held-out code PPL |

| --- | ---: | ---: |

| Q4_0 reference | about 4.6 | 2.4775 |

| Optimized mixed Q2_0/Q4_0 | 2.47 | 2.8204 |

| Optimized mixed Q2_0/Q4_0 | 2.97 | 2.7291 |

| Earlier stock-Q2_0/Q4_0 experiment (not published) | 3.52 | 3.8256 |

These short-run figures are screening results rather than a full benchmark suite. The optimized low-bit files produced coherent deterministic arithmetic, code-generation, and formatting results, but both answered one trick-question smoke incorrectly ("all but 9" as eight rather than nine). They remain labeled experimental because aggressive quantization can reduce capability unevenly.

Usage

hf download vcruz305/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF \
  Nemotron-3.5-Lightning-30B-A3B-Q4_0.gguf \
  --local-dir .

./llama-cli \
  -m Nemotron-3.5-Lightning-30B-A3B-Q4_0.gguf \
  -ngl 999 \
  -c 8192

Increase context carefully; this model supports a very large native context, but KV-cache memory grows with the configured context length.

Credits and license

• Base model and architecture: NVIDIA.
• Quantization and importance matrix: vcruz305, using llama.cpp.
• License: OpenMDW-1.1, following the base model. Review the upstream model card and license before use.

This is a community redistribution and is not affiliated with NVIDIA.