---

license: other

license_name: nvidia-open-model-license

license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/

base_model: nvidia/NVIDIA-Nemotron-Nano-9B-v2

tags:

• gguf
• nemotron_h
• mamba2
• hybrid
• ssm
• xyntetik-runner

pipeline_tag: text-generation

---

NVIDIA Nemotron-Nano-9B-v2 — Q8_0 GGUF (canonical, xyntetik-runner)

<!-- BEGIN runner-toolcalling -->

Serve with xyntetik-runner &mdash; tool calls that still parse when the budget runs out

This GGUF is built to be served with xyntetik-runner, a single-binary, dependency-free C11 inference engine (CPU / CUDA / Metal, OpenAI-compatible server). Its one differentiator you cannot get from another engine:

> When a tool call runs past its token budget, Runner closes it to the smallest schema-legal document, so the arguments still parse.

Most OpenAI-compatible engines return finish_reason: "length" with an empty or malformed tool_calls the caller has to repair or retry. Runner does forced-truncation recovery: once a JSON document starts, it emits a legal ending when the budget expires. On local models &mdash; tight context, slow generation &mdash; that is the difference between an agent loop that finishes and one that retries from scratch.

Measured &mdash; what each engine hands the caller when the token budget cuts a tool call short (same box, same tool schema, same prompt, tool_choice:"required", temperature 0, budgets 1&rarr;64):

| engine | budget too small (1&ndash;16 tokens) | enough budget (64, control) |

|---|---|---|

| Runner | executable tool_calls, arguments parse | completes |

| vLLM 0.27.1 | no call; protocol framing leaks into content | completes |

| llama.cpp b10488 | no call; leak, then tool_calls with unparseable args | completes |

| Ollama 0.32.14 | no call; empty content, then HTTP 500 | completes |

| TensorRT-LLM 1.2.1 &dagger; | no call; <tool_call> leak, then empty content | completes |

| SGLang 0.5.17 &dagger; | no call; <tool_call> leak, then empty content | completes |

Every engine completes at 64 tokens &mdash; the failure is truncation, not misconfiguration. Below that, only Runner returns an executable call. This is the behaviour across every OpenAI-compatible engine measured, not a claim about engines not measured. &dagger; TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute; truncation recovery is a property of the runtime, not the model.

Constrained decoding also holds the shape of a tool call under quantization: on a full quant ladder, schema conformance and tool selection stayed 100% down to Q4_0 (argument agreement decayed to 50%) &mdash; it guarantees the shape of a call at any quantization, not its contents.

Quickstart

# build the single binary (no deps), then serve this GGUF:
runner -m NVIDIA-Nemotron-Nano-9B-v2-Q8_0.gguf --serve
# point any OpenAI-compatible client at http://localhost:8080/v1

Full recipe, raw responses, and the per-release regression gate: the truncation benchmark.

<!-- END runner-toolcalling -->

Runs on xyntetik-runner (nemotron_h hybrid)

Nemotron-Nano-9B-v2 is a hybrid Mamba-2 / attention model — architecture nemotron_h, a state-space (SSM) + attention hybrid. This GGUF loads and decodes on xyntetik-runner (CPU / CUDA) and on stock llama.cpp (verified against b10353).

runner -m NVIDIA-Nemotron-Nano-9B-v2-Q8_0.gguf --serve

Canonical, interoperable Q8_0

This file was produced by xyntetik-runner --quantize --quant q8_0 from the bf16 GGUF base. After a quantizer fix (efb041a), runner's Q8_0 tensor data is byte-for-byte identical to what llama.cpp's own Q8_0 quantizer produces from the same input — all 122 quantized tensors and all 219 F32 tensors match exactly. Runner emits canonical ggml Q8_0: a runner-produced Q8_0 and a llama.cpp-produced Q8_0 of the same input are the same weights (only container metadata KV ordering differs).

Fidelity vs the bf16 base — measured, not asserted

Two independent CPU/greedy checks against the bf16 base as ground truth:

Next-token fidelity — 400-position KLD + top-1 agreement over a pinned corpus (top-20 logprob surface, scripts/kld-compare.py):

| Q8_0 file | mean KLD vs bf16 | top-1 agreement | top-8 overlap |

|---|---|---|---|

| this file (canonical runner Q8_0) | 2.46e-4 | 100.0% | 97.7% |

| llama.cpp Q8_0 (bartowski) | 2.46e-4 | 100.0% | 97.7% |

| pre-fix runner Q8_0 (non-canonical) | 2.41e-4 | 100.0% | 97.8% |

All three are within noise of each other and agree with bf16 on the top-1 token 100% of the time — Q8_0 is effectively lossless here. The canonical file's KLD is identical to llama.cpp's (byte-identical weights). The pre-fix, non-canonical runner file was not lower quality — it was merely bit-different from ggml; the fix is an interoperability / canonicality win, not a quality change.

Server-path greedy identity (2026-08-20, reproducible) — via the committed scripts/reference_compare.py (both engines served, exact generated UTF-8 from /v1/completions at temperature 0; 5 prompts × 32 tokens, CPU): 5/5 byte-identical vs llama.cpp ea12b27 — factual, code, counting, story, and JSON prompts all match exactly. Raw JSON: ssm-greedy-reference-2026-08-20/.

Byte-identity spot-check — 6-prompt greedy vs the bf16 base, on both engines: this Q8_0 scores 5/6 on runner and 5/6 on llama.cpp. The one miss is a near-tie argmax coin-flip (identical prefix, one differing word choice) — and the canonical llama.cpp Q8_0 shows the same miss. This is a near-tie byte-identity check, not a quality measurement.

An envelope manifest (NVIDIA-Nemotron-Nano-9B-v2-Q8_0.gguf.envelope.json) is included. Verdict: experimental — a dated measurement, not a standing quality claim (no per-model conformance gate report).

Model, attribution, and license

• Base model: nvidia/NVIDIA-Nemotron-Nano-9B-v2, a hybrid Mamba-2 / attention (nemotron_h) model by NVIDIA. All model weights, capabilities, and credit belong to NVIDIA; this repository only redistributes a quantized GGUF conversion of NVIDIA's weights.
• License: NVIDIA Open Model License. Use of this model is governed by the NVIDIA Open Model License; by using these weights you agree to its terms.
• Quantization: bf16 GGUF → Q8_0 via xyntetik-runner --quantize --quant q8_0 (canonical ggml Q8_0). GGUF sha256 d3224286484df1ae79c2b74d2a0913e1bbb5a364b0dd7c1061c95fa97ee608d7, ~8.81 GiB (9,458,093,632 bytes).
• Not affiliated with, sponsored by, or endorsed by NVIDIA.