---

license: other

license_name: nvidia-open-model-license

license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/

base_model: nvidia/NVIDIA-Nemotron-Nano-9B-v2

tags:

- nemotron_h

- text-generation

- gguf

- bf16

- q4_k_m

- llama.cpp

- nvidia

- nemotron

pipeline_tag: text-generation

inference: false

model_type: nemotron_h

---

NVIDIA-Nemotron-Nano-9B-v2 (GGUF)

<a href="https://console.runpod.io/hub/template/m7g1jafoep?ref=97dt77nr">

<img src="https://img.shields.io/badge/Deploy%20on%20RunPod-16GB%20GPU-8A2BE2" alt="Deploy on RunPod">

</a>






GGUF conversions of nvidia/NVIDIA-Nemotron-Nano-9B-v2,

a general-purpose instruction-tuned reasoning model from NVIDIA's Nemotron

family, converted for use with llama.cpp.

Two quantizations are provided:

| Quantization | File | Size | Use case |

|--------------|------|------|----------|

| bf16 | NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf | 17.79 GB | Full precision — maximum quality, needs 24 GB+ VRAM |

| Q4_K_M | NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf | 6.53 GB | Fits entirely on a single 16 GB GPU with ~9 GB left for KV cache |

Model Summary

| Property | Value |

|----------|-------|

| Base model | nvidia/NVIDIA-Nemotron-Nano-9B-v2 |

| Architecture | NemotronHForCausalLM (nemotron_h) |

| Parameters | ~9B |

| Layers | 56 |

| Attention heads | 40 (8 KV heads) |

| Hidden size | 4480 |

| Context length | 131,072 |

| Vocab size | 131,072 |

| Reasoning | Yes (thinking mode, /think & /no_think controls) |

| Tool calling | Native (<tool_call> XML format) |

Files

impacte/NVIDIA-Nemotron-Nano-9B-v2-GGUF/
├── NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf     # bf16 GGUF (17.79 GB)
├── NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf   # Q4_K_M GGUF (6.53 GB)
└── .gitattributes                            # LFS tracking

Usage

llama.cpp (local inference)

# bf16 (full quality, 24 GB+ VRAM)
llama-server \
  -m NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf \
  --ctx-size 8192 \
  --port 8080

# Q4_K_M (single 16 GB GPU)
llama-server \
  -m NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf \
  --ctx-size 32768 \
  --port 8080

Then call the OpenAI-compatible endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M",
    "messages": [
      {"role": "user", "content": "Explain what a Tauri v2 app is."}
    ]
  }'

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,  # offload all layers to GPU
)

About the base model

NVIDIA-Nemotron-Nano-9B-v2 is a compact, instruction-tuned reasoning model in

NVIDIA's Nemotron family, trained on NVIDIA's post-training and pretraining

datasets. It supports multiple languages (en, es, fr, de, it, ja) and is

designed for efficient local deployment with a hybrid Mamba-2 + attention

architecture.

License & Attribution

• Base model: nvidia/NVIDIA-Nemotron-Nano-9B-v2 — released under the

NVIDIA Open Model License. Review before commercial use.

• GGUF conversion: performed with llama.cpp's convert_hf_to_gguf.py (bf16)

and llama-quantize (Q4_K_M).

> Note: This is a GGUF conversion of a model under the NVIDIA Open Model

> License. Ensure your use complies with that license.

Limitations

• The bf16 GGUF (17.79 GB) preserves full precision but is large — needs

24 GB+ VRAM.

• The Q4_K_M GGUF (6.53 GB) fits on a single 16 GB GPU but trades some

precision for size.

• The base model's general capabilities are retained; this is a direct

conversion with no additional fine-tuning.