---

license: other

license_name: nvidia-open-model-license

license_link: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B

base_model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- moe

- apex

- quantized

- llama.cpp

- mamba2

- hybrid

- nemotron

---

Nemotron-3-Super-120B-A12B — APEX quants

APEX (measured/structural per-tensor bit allocation) GGUF quants of

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B

— 120B total / ~12B active. Architecturally the most unusual model this pipeline has handled: a

hybrid backbone where each of 88 layers is exactly one of three mixer types (never a

combination) — 8 pure-Attention layers, 40 pure-Mamba2 (linear-attention/SSM) layers, and 40

MoE-FFN layers whose experts route through a shared low-rank latent bottleneck (NVIDIA's

"LatentMoE") rather than a standard SwiGLU gate.

Status: two tiers available in this upload (i-quality, i-compact — see note below on

i-mini). Real-generation coherence + chained tool-calling (dependent 3-turn chain + a distractor

tool that must never be called) both pass 3/3 — see GATE.md. No PPL-ratio: the true

bf16 reference is ~240GB and doesn't fit resident on the build hardware to run a baseline pass.

⚠️ Set --ctx-size explicitly — do not run this model with defaults

Nemotron-3-Super was trained with up to a 1,048,576-token (1M) context window. If you launch

llama-cli/llama-server without an explicit --ctx-size, llama.cpp defaults the KV cache to

the model's own trained context length, not a small sane default — an attempt to allocate a KV

cache sized for a million tokens, which can consume very large amounts of memory and stall or

crash a machine with limited RAM/VRAM.

Always pass --ctx-size sized to what you actually need, e.g. --ctx-size 8192 for typical

chat/tool-use. Only reach for six-figure-plus context sizes if you have the RAM/VRAM to back it.

Sizes

| size | routed-expert precision | file size | bits/weight | coherent | tool calls |

|---|---|---|---|---|---|

| i-quality | Q6_K (edge/near bands), Q4_K (middle) | 85.78 GB | 5.10 | yes | 3/3 |

| i-compact | Q5_K/Q4_K (edge/near), Q3_K (middle) | 67.82 GB | 4.02 | yes | 3/3 |

i-mini isn't part of this upload round. Its planned routed-expert middle band targeted

IQ2_XXS, but moe_intermediate_size=2688 isn't divisible by 256, so llama-quantize can't apply

that i-quant type there and silently falls back to a higher-precision type for those tensors

(WARNING: 40 of 763 tensor(s) required fallback quantization) — landing at 3.81 bpw / 53.6 GB

instead of the ~2.8 bpw / 42 GB originally targeted. Worth remembering for any future recipe on

this architecture.

Attention, the two Mamba2 projections (ssm_in/ssm_out), and the shared latent bottleneck stay

at higher precision (Q6_K–Q8_0) in every tier; all state-space scalars/norms (ssm_conv1d,

ssm_a, ssm_d, ssm_dt.bias, ssm_norm.weight) and the router stay F32, same as the reference

community quant.

Method

Built without ever downloading the model's raw safetensors at inference precision twice over. The

imatrix was computed by a new band-serialized generator

(serialized_gen_nemotron.py/imatrix_sources.py's nemotron_h_moe arch adapter) that drives the

real transformers NemotronHBlock module — materializing one layer's weights at a time from a

full-precision BF16 GGUF and hooking its real forward pass — rather than hand-reimplementing the

Mamba2 selective-scan recurrence, which would be far more likely to have a subtle bug than driving

the library's own tested module. Three GGUF-side tensor transforms (A_log = log(-ssm_a),

D/ssm_norm.weight reshapes, conv1d.weight unsqueeze) were verified against llama.cpp's own

conversion source before trusting them, not guessed.

Bit allocation is structural (role/depth-band priors) — this arch doesn't have a measured-

sensitivity map yet (would need a memory-bounded redesign of that mode, deferred for now).

Calibration

Imatrix computed over a large, deliberately diverse local corpus (~2MB: encyclopedic/wikitext

prose, source code in several languages, classic literature, and technical/scientific text), 126

chunks × 512 tokens. Of the 40,960 total (expert × up/down-tensor) rows across all MoE layers,

only 8 never saw a routed token across the whole corpus — strong coverage of the full 512-expert

population.

Attribution & licenses

• Base: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B (NVIDIA Nemotron Open Model License — see base model repo for terms)
• BF16 source: unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF (@unsloth) — full-precision BF16 GGUF tier, used as the imatrix source so the raw safetensors never needed downloading
• Engine: llama.cpp (MIT)
• APEX: localai-org/apex-quant (@mudler) (MIT)

Unofficial community quantization; not affiliated with or endorsed by NVIDIA. Use of this model is

subject to the NVIDIA Nemotron Open Model License linked above.