---

base_model:

• nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

language:

• en
• es
• fr
• de
• it
• ja

library_name: transformers

license: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

pipeline_tag: text-generation

tags:

• nvidia
• nemotron-3.5
• gguf
• llama.cpp
• text-generation
• moe

---

NVIDIA-Nemotron-3.5-Lightning-30B-A3B - GGUF

This repository contains GGUF format model files for NVIDIA's NVIDIA-Nemotron-3.5-Lightning-30B-A3B.

These files were converted and quantized using llama.cpp.

Available Files

| Filename | Quant Method | Description |

| --- | --- | --- |

| nemotron-3.5-lightning-30b-a3b-bf16.gguf | BF16 | Full-precision reference weights (unquantized) |

| nemotron-3.5-lightning-30b-a3b-Q8_0.gguf | Q8_0 | Extremely high quality, fast, high resource usage |

| nemotron-3.5-lightning-30b-a3b-Q6_K.gguf | Q6_K | Very high quality, near-lossless quantization |

| nemotron-3.5-lightning-30b-a3b-Q5_K_M.gguf | Q5_K_M | High quality, balanced performance and memory |

| nemotron-3.5-lightning-30b-a3b-Q5_K_S.gguf | Q5_K_S | High quality, slightly smaller footprint than Q5_K_M |

| nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf | Q4_K_M | Recommended balance of size, speed, and quality |

| nemotron-3.5-lightning-30b-a3b-Q4_K_S.gguf | Q4_K_S | 4-bit quantization with small memory footprint |

| nemotron-3.5-lightning-30b-a3b-Q4_0.gguf | Q4_0 | Standard 4-bit quantization |

| nemotron-3.5-lightning-30b-a3b-dflash.gguf | — | DFlash speculative decoding draft model |

Model Summary

| Total Parameters | 30B (3B active) |

| --- | --- |

| Architecture | MoE — Mamba-2 + MoE + Attention hybrid |

| Context Length | Up to 1M tokens (256K native default) |

| Supported Languages | English (and coding languages), Spanish, French, German, Italian, Japanese |

| Speculative Decoding | DSpark, DFlash, MTP (Multi-Token Prediction) |

| Reasoning Mode | Configurable on/off via chat template (enable_thinking=True/False) |

| Recommended Sampling | Temperature 1.0, Top_P 0.95 |

| License | OpenMDW License Agreement, version 1.1 |

| Original Model | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 |

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The pre-training data has a cutoff date of September 2025.
• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.

Description

NVIDIA-Nemotron-3.5-Lightning-30B-A3B is a large language model (LLM) trained by NVIDIA.

The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. The Lightning 3.5 model is released alongside speculative decoding methods (DSpark, DFlash, MTP) for faster text generation. The model has 3B active parameters and 30B parameters in total.

This model is ready for commercial use under the OpenMDW-1.1 license.

Usage with llama.cpp

CLI / llama-cli

Reasoning ON (default):

llama-cli \
  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
  --jinja \
  --chat-template-file chat_template.jinja \
  -p "Write a Python function to compute Fibonacci numbers." \
  --temp 1.0 --top-p 0.95 \
  -ngl 99

llama-server

Start the OpenAI-compatible server:

llama-server \
  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
  --temp 1.0 --top-p 0.95 \
  -np 1 \
  -c 40960 \
  --port 8000 \
  -ngl 99 \
  -fa on \
  --jinja \
  --chat-template-file chat_template.jinja \
  --no-webui \
  --fit off

With DFlash Speculative Decoding

Accelerate token generation using the DFlash draft model:

llama-server \
  -m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
  -md nemotron-3.5-lightning-30b-a3b-dflash.gguf \
  --draft-max 6 \
  --temp 1.0 --top-p 0.95 \
  -np 1 \
  -c 40960 \
  --port 8000 \
  -ngl 99 \
  -ngld 99 \
  -fa on \
  --jinja \
  --chat-template-file chat_template.jinja \
  --no-webui \
  --fit off

API Client Example (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

# Reasoning ON (default)
response = client.chat.completions.create(
    model="nemotron-3.5-lightning-30b-a3b",
    messages=[{"role": "user", "content": "Write a haiku about GPUs"}],
    max_tokens=4096,
    temperature=1.0,
    top_p=0.95,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.content)

# Reasoning OFF (direct answer)
response = client.chat.completions.create(
    model="nemotron-3.5-lightning-30b-a3b",
    messages=[{"role": "user", "content": "What is the capital of Japan?"}],
    max_tokens=128,
    temperature=1.0,
    top_p=0.95,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Benchmarks

Reasoning Benchmark Evaluations

| Task | Nemotron-3.5-Lightning-30B-A3B-BF16 | Nemotron-3.5-Lightning-30B-A3B-NVFP4 |

| --- | --- | --- |

| General Knowledge | | |

| MMLU Pro | 81.94 | 81.62 |

| AA-Omniscience | 17.50 | 16.63 |

| Reasoning | | |

| GPQA Diamond (no tools) | 75.44 | 75.57 |

| HLE (text-only, no tools) | 11.72 | 10.47 |

| SciCode | 32.60 | 31.38 |

| Coding & Agentic | | |

| SWE-bench Verified | 51.56 | 52.80 |

| SWE-bench Multilingual | 39.33 | 36.47 |

| Terminal-Bench 2.1 | 24.58 | 23.46 |

| PinchBench | 85.37 | 83.43 |

| BrowseComp | 36.97 | 36.81 |

| τ³-bench (Banking) | 9.28 | 9.48 |

| GDPval-AA-V2 | 832 | 865 |

| Instruction Following | | |

| IFBench (loose) | 71.88 | 72.88 |

| Long Context | | |

| AA-LCR | 52.00 | 49.19 |

License and Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW-1.1 model license.