---

pipeline_tag: text-generation

base_model:

• nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

base_model_relation: quantized

license: other

license_name: nvidia-nemotron-open-model-license

license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/

library_name: llama-cpp

tags:

• nvidia
• nemotron
• ModelOpt
• quantized
• NVFP4
• nvfp4
• nemotron3
• nemotron-3-nano-30b-a3b

language:

• en
• es
• fr
• de
• ja
• it

---

Description

Nemotron 3 Nano 30B A3B NVFP4 GGUF created from https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4

Model Overview

Nemotron-3-Nano-30B-A3B is a large language model (LLM) trained from scratch by NVIDIA, designed as a unified model for both reasoning and non-reasoning tasks. It responds to user queries and tasks by first generating a reasoning trace and then concluding with a final response. The model's reasoning capabilities can be configured through a flag in the chat template. If the user prefers the model to provide its final answer without intermediate reasoning traces, it can be configured to do so, albeit with a slight decrease in accuracy for harder prompts that require reasoning. Conversely, allowing the model to generate reasoning traces first generally results in higher-quality final solutions to queries and tasks.

The model employs a hybrid Mamba2-Transformer Mixture-of-Experts (MoE) architecture, consisting of 23 Mamba-2 and 23 MoE layers, along with 6 Attention layers. Each MoE layer includes 128 routed experts plus 1 shared expert, with 6 experts activated per token. The model has 3.5B active parameters and 30B parameters in total. Supported languages include English, German, Spanish, French, Italian, and Japanese.

This repository contains a GGUF conversion of the NVIDIA NVFP4-quantized checkpoint, making it usable with llama.cpp and other GGUF-compatible inference engines.

Model Architecture

| Property | Value |

| :---------------------- | :----------------------------------------- |

| Architecture Type | Mamba2-Transformer Hybrid MoE |

| Total Parameters | 30B |

| Active Parameters | 3.5B |

| Layers | 52 (23 MoE + 23 Mamba-2 + 6 Attention) |

| Context Length | 256K tokens (up to 1M supported) |

| Vocabulary Size | 131,072 |

| MoE Configuration | 6 active / 128 routed experts + 1 shared |

| Attention Heads | 32 (2 KV heads, Grouped Query Attention) |

| Hidden Size | 2688 |

| Quantization | NVFP4 (KV cache: FP8) |

| Supported Languages| English, Spanish, French, German, Japanese, Italian |

GGUF File

| File | Description |

| :--------------------------------- | :------------------------------------------ |

| nemotron-3-nano-30b-a3b-nvfp4.gguf | Single-file GGUF model (NVFP4 quantization) |

A chat_template.jinja file is also provided for use with chat-based inference.

Usage

llama.cpp

# Build llama.cpp with CUDA support (recommended for NVFP4)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Run inference
./build/bin/llama-cli \
  -m nemotron-3-nano-30b-a3b-nvfp4.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 1.0 --top-p 1.0

llama-server (OpenAI-compatible API)

./build/bin/llama-server \
  -m nemotron-3-nano-30b-a3b-nvfp4.gguf \
  --host 0.0.0.0 --port 8080

Generation Parameters

The recommended generation parameters (from the original model's generation_config.json):

| Parameter | Value |

| :------------ | :--------- |

| Temperature | 1.0 |

| Top-P | 1.0 |

| BOS Token ID | 1 |

| EOS Token IDs | 2, 11 |

| Pad Token ID | 0 |

> For reasoning tasks: temperature=1.0, top_p=1.0. For tool calling: temperature=0.6, top_p=0.95.

Evaluation Results (copied from: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4)

| Benchmark | BF16 | FP8 | NVFP4 |

| ---------------------------- | ----- | ----- | ----- |

| MMLU-Pro | 78.3 | 78.1 | 77.4 |

| AIME25 (no tools) | 89.1 | 87.7 | 86.7 |

| GPQA (no tools) | 73.0 | 72.5 | 71.9 |

| LiveCodeBench (v6) | 68.3 | 67.6 | 65.4 |

| SciCode (subtask) | 33.0 | 31.9 | 30.7 |

| HLE (no tools) | 10.2 | 10.3 | 9.4 |

| TauBench V2 (Average) | 49.0 | 47.0 | 45.6 |

| IFBench (prompt) | 71.5 | 72.2 | 70.7 |

| AA-LCR | 35.9 | 36.1 | 33.3 |

| MMLU-ProX (avg over langs) | 59.50 | 59.6 | 57.8 |

> Baseline: NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

> Benchmarked with temperature=1.0, top_p=1.0

Acknowledgements

• Original model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
• NVFP4 quantization: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 (via NVIDIA Model Optimizer)

License

NVIDIA Nemotron Open Model License