---

license: other

license_name: nvidia-open-model-license

license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/

base_model: nvidia/Nemotron-H-8B-Base-8K

tags:

- gguf

- turboquant

- tq4_1s

- config-i

- nemotron-h

- mamba2

- hybrid

- quantized

---

Nemotron-H-8B-Base-8K — Config-I (TQ4_1S) GGUF

First Config-I / tq4_1s quantization of a Nemotron-H (Mamba-2 hybrid) model.

This is a TurboQuant Config-I weight quantization

(TQ4_1S — RHT/Walsh-Hadamard-rotated 4-bit, 16-level Lloyd-Max codebook, dual half-block scales, ~5.0 bpw)

of NVIDIA's nvidia/Nemotron-H-8B-Base-8K — a

hybrid Mamba-2 / attention language model (52 layers: interleaved Mamba-2 SSM, self-attention, and

squared-ReLU FFN blocks). Config-I / tq4_1s had previously only been validated on dense and MoE

Transformer (Qwen) models — **this is its first application to the Nemotron-H / Mamba-2-hybrid

architecture family.**

Sizes

| Variant | Size | vs BF16 | vs Q8_0 |

|---|---:|---:|---:|

| BF16 (source GGUF) | 16.2 GB | — | — |

| Q8_0 (baseline) | 8.0 GB | −51% | — |

| Config-I (tq4_1s) | 4.8 GB | −70% | −40% |

At 4.8 GB the whole model fits comfortably in the memory of a single consumer GPU / NVIDIA DGX Spark

(GB10) / RTX 5090, with plenty of headroom for context.

Quality — perplexity (wikitext, ctx 512)

| Variant | PPL | Δ vs Q8_0 |

|---|---:|---:|

| Q8_0 (baseline) | 6.8211 | — |

| Config-I (tq4_1s) | 7.0190 | +0.198 (+2.9%) |

A 2.9% perplexity increase for a 40%-smaller file (4-bit vs 8-bit) — the SSM recurrence

(A/Δt/conv/D), attention, and squared-ReLU FFN all survive the rotated-4-bit quantization intact.

This is, to our knowledge, the first evidence that TurboQuant's RHT-rotated 4-bit format transfers

cleanly to a Mamba-2-hybrid architecture, not just to Transformers.

Intended use & limitations

This is a base (pretrained) model — completion / few-shot only. It has no chat template and is

not instruction-tuned; prompt it with plain text continuations (e.g. "The capital of France is"), not

chat turns. Context length is 8K (Nemotron-H-8B-Base-8K). Quantization is lossy: expect the +2.9%

perplexity shown above vs the 8-bit baseline. eos_token_id is [2, 11] — pass both if you want

generation to stop on either.

Running it

tq4_1s is a custom TurboQuant weight type and Nemotron-H is a hybrid Mamba-2 architecture, so this

GGUF requires the TheTom/llama-cpp-turboquant fork

(which implements both TQ4_1S decode and the nemotron_h / nemotron_h_moe inference graphs).

Stock llama.cpp will not load it.

llama-cli -m Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf -p "The capital of France is" -n 64

Backend notes

• CUDA (e.g. NVIDIA DGX Spark GB10, RTX 50-series) — the intended runtime, and **verified on a

DGX Spark (GB10): PPL 7.0037**, coherent generation at ~24 tok/s decode / ~50 tok/s prefill, the

whole 4.9 GB fitting comfortably alongside other resident models. TQ4_1S decodes via a

warp-cooperative dequant-to-float path that applies the inverse-RHT before the matmul, so the

default path is numerically exact (no flags needed).

• CPU — supported; verified (PPL 7.0190 on the CPU dequant-to-float path).
• Metal (Apple Silicon) — supported with the environment flag TQ_NO_ROTATE=1, e.g.

TQ_NO_ROTATE=1 llama-cli -m Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf -p "...". Verified PPL

7.0072 (matches CPU/CUDA to fp16 noise). The fork's default fused "rotate-act" TQ matmul

(which pre-rotates activations in-place and uses a no-RHT weight dequant) produces NaN on

Nemotron-H's squared-ReLU FFN on Metal; TQ_NO_ROTATE=1 routes TQ weights through the standard

mul_mm with the exact inverse-RHT dequant instead. Transformer (Qwen) models are unaffected on the

default path.

Reproduction

# 1. convert the HF safetensors to a bf16 GGUF (needs a converter with NemotronHForCausalLM support)
python convert_hf_to_gguf.py Nemotron-H-8B-Base-8K --outtype bf16 --outfile nemoh8b-bf16.gguf
# 2. Config-I quantize with the fork's llama-quantize
llama-quantize nemoh8b-bf16.gguf Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf TQ4_1S
# (Q8_0 baseline for comparison: llama-quantize nemoh8b-bf16.gguf nemoh8b-Q8_0.gguf Q8_0)
# 3. perplexity check
llama-perplexity -m Nemotron-H-8B-Base-8K-Config-I-tq4_1s.gguf -f wikitext.txt -c 512

Attribution

• Base model: © NVIDIA, Nemotron-H-8B-Base-8K,

under the NVIDIA Open Model License.

• Quantization method: TurboQuant

(RHT-rotated extreme weight compression) + the tq4_1s / Config-I weight format from the

llama-cpp-turboquant fork.