---

base_model: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16

base_model_relation: quantized

quantized_by: AtomicChat

pipeline_tag: image-text-to-text

library_name: gguf

license: other

tags:

• atomic-dynamic
• gguf
• llama.cpp
• nvidia
• imatrix
• quantized
• conversational
• atomic-chat

---

NVIDIA Nemotron 3.5 Super VL 120B-A12B · Atomic Dynamic GGUF

<p style="margin-top: 0; margin-bottom: 0;">

<em><b>Atomic Dynamic</b> (AD) quants, built from NVIDIA's original BF16 weights with our own importance matrix. Every file here matches or beats the other GGUF builds of the same size or larger, measured side by side on one stand. The <a href="https://huggingface.co/datasets/AtomicChat/calib-corpora">calibration corpora</a> behind our builds are public.</em>

</p>

<div style="display: flex; gap: 8px; align-items: center; margin-top: 10px; margin-bottom: 10px;">

<a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_nvidia_nemotron_3_5_super_vl_120b_a12b&utm_content=btn_atomic"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_atomic.png" width="162" alt="Atomic Chat"></a>

<a href="https://discord.gg/8wGSsvmg4V"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_discord.png" width="119" alt="Discord"></a>

<a href="https://github.com/AtomicBot-ai/Atomic-Chat"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_github.png" width="115" alt="GitHub"></a>

</div>

<ul style="margin: 0 0 12px 0;">

<!-- TODO(editorial): guide bullet: link to the atomic.chat blog guide for this model, if there is one -->

<li>Every file here was measured against the BF16 original on the same stand as the other publishers' files, and each one beats or ties the file of theirs that is the same size or larger.</li>

<li>See our quantization analysis below for measurements and instructions.</li>

</ul>

<hr style="margin: 0 0 16px 0;">

<img src="https://huggingface.co/AtomicChat/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF/resolve/main/atomic-dynamic-kld.png" alt="Atomic Dynamic vs other GGUF builds: mean KL divergence to the BF16 original against size on disk" width="100%">

Lower is closer to the original. Each purple point is a file in this repo; the grey

ones are the other publishers' files, downloaded and measured with exactly the

same reference, text and llama.cpp build. At 75 GB, AD-IQ4_NL drifts about half as

much as both NVFP4 builds, which are the same size or up to 3 GB larger. At 70 GB,

AD-IQ3_M drifts about half as much as LM Studio's Q4_K_M of the same size. At

65 GB, AD-IQ3_XXS drifts 20% less than Unsloth's UD-Q3_K_XL, and from 46 to 58 GB

the AD-Q2_0_* files drift 6 to 18% less than the Unsloth file of each size, and at

87 GB AD-Q4_K_S drifts 13% less than UD-Q4_K_XL.

What is Atomic Dynamic

Atomic Dynamic is how we decide which tensor gets which bits. The files use only

standard llama.cpp quantization types: upstream llama.cpp with nemotron_h_moe

support reads them, no fork and no special build.

• The experts take the compression, nothing else does. In this model about 93%

of the weights are experts. The Mamba-2 projections, attention, the latent

projections and the shared expert together are under 6% of the size, so keeping

them at q8_0 or q6_k in every file costs little, and every bit saved goes to

the experts instead.

• Bits follow the measured activity, not the position. The blocks that get

extra precision are the ones the importance matrix ranks highest (here 76, 79,

81, 83, 85, 87), not a fixed fraction of the depth.

• Calibrated on the real chat format. The importance matrix is collected on the

BF16 weights over 4.6M tokens rendered through this model's own chat template,

with the special tokens read as tokens.

• A better q2_0. llama.cpp's own q2_0 quantizer ignores the importance matrix

and scales each block by its largest weight, so it never uses the format's fourth

level and rounds most weights to zero. Ours searches each block's scale over both

signs, uses all four levels and weighs the error by the importance matrix: on

this model the same 51 GB layout drifts 0.107 with it and 0.258 with the stock

quantizer. The files are ordinary q2_0, so nothing special is needed to run them.

• No silent type swaps. llama.cpp quietly replaces types that do not fit a

tensor's shape. We pick the replacement ourselves and check every tensor's type

in the finished file.

• Measured against everyone, published only where we win. Each file is set

against the other publishers' files of the same size or larger before it goes up.

Pick a file

Every number below is measured, not estimated. How we measured it is at the

bottom, and the raw logs are in

the metrics repo

so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the

original weights. Lower is better, and zero means identical. top-1 is how

often it picks the same next word the original would have picked.

| File | Size | KL divergence | top-1 |

|---|---:|---:|---:|

| AD-Q8_0 | 131.6 GB | 0.00442 | 97.22% |

| AD-Q6_K | 120.0 GB | 0.00476 | 97.13% |

| AD-Q5_K_S | 110.6 GB | 0.00603 | 96.76% |

| AD-Q4_K_S | 86.9 GB | 0.00816 | 96.23% |

| AD-IQ4_NL | 74.8 GB | 0.01327 | 95.24% |

| AD-IQ3_M | 70.0 GB | 0.02366 | 93.75% |

| AD-IQ3_XXS | 65.3 GB | 0.03523 | 92.43% |

| AD-IQ2_M | 59.3 GB | 0.06429 | 89.75% |

| AD-Q2_0_XL | 57.4 GB | 0.06566 | 90.02% |

| AD-Q2_0_L | 55.8 GB | 0.07219 | 89.44% |

| AD-Q2_0_M | 51.0 GB | 0.10652 | 86.97% |

| AD-Q2_0_S | 45.7 GB | 0.14704 | 84.86% |

The AD-Q2_0_* files put q2_0 on the expert down projections of the least

active blocks, quantized with our own importance-weighted q2_0 quantizer (below).

AD- marks an Atomic Dynamic layout: the experts are compressed hardest, the

attention, Mamba, shared-expert and embedding tensors stay at q8_0 or q6_k, and the

blocks the importance matrix ranks highest get more bits. The rest of the name is

the type of the expert up projections, the largest tensor group (the AD-Q2_0_*

files are named after the q2_0 on their down projections instead); the exact type of

every tensor is in ladder/ in the metrics repo. Compare files by size on disk,

not by name.

Which one fits your card

The file has to fit, and so does the context. Only 8 of the 88 layers are

attention; the Mamba layers keep a fixed-size state. The attention cache is about

8 KB per token, 0.1 GB at 8k context and 0.2 GB at 32k, plus about 1 GB of

compute buffers.

| Memory for the model (GPU or unified) | Largest file that fits |

|---|---|

| 48 GiB | AD-Q2_0_S (45.7 GB on disk) |

| 56 GiB | AD-Q2_0_M (51.0 GB on disk) |

| 64 GiB | AD-IQ3_XXS (65.3 GB on disk) |

| 80 GiB | AD-IQ4_NL (74.8 GB on disk) |

| 96 GiB | AD-Q4_K_S (86.9 GB on disk) |

| 128 GiB | AD-Q6_K (120.0 GB on disk) |

| 192 GiB and up | AD-Q8_0 (131.6 GB on disk) |

> [!TIP]

> If you are choosing between two neighbouring files, take the larger one. The

> steps between them cost one or two gigabytes and buy noticeably fewer wrong words.

Running it

llama-server -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-AD-IQ3_M.gguf -ngl 99 -c 8192

The chat template is inside every file, llama.cpp applies it on its own.

<!-- TODO(editorial): prompt format block and the thinking toggle, if the family has one -->

The model's multi token prediction head is inside every file (q8_0 in AD-Q8_0

and AD-Q6_K, q5_k down to AD-IQ3_XXS, q4_k below). We have not tested speculative decoding with it yet;

the KL divergence numbers do not involve it.

Speed

<!-- TODO(editorial): llama-bench prompt and generation t/s at 8k and 32k for the recommended file -->

Images

This model reads images, not only text. That needs one extra file, the vision

projector, which is separate from the quant and shared by all of them:

• mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-F16.gguf for most people
• mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf if your setup prefers bfloat16

Download it once alongside whichever quant you picked:

llama-mtmd-cli \
  -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-AD-IQ3_M.gguf \
  --mmproj mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-F16.gguf \
  --image your-photo.jpg -ngl 99 -c 8192 \
  -p "What is in this image?"

We checked image input on AD-Q8_0 with a simple shapes-and-colours picture.

Video input was not tested.

<!-- TODO(editorial): family specific vision flags (e.g. --image-min-tokens) and the demo image test -->

How these compare to other builds

We downloaded the other publishers' files and measured them with the same

reference logits, held-out text, context, chunk count and llama.cpp commit as

ours. Each of our files is set against the other file that is the same size or

larger:

| Ours | Size | KL divergence | top-1 | Theirs | Size | KL divergence | top-1 |

|---|---:|---:|---:|---|---:|---:|---:|

| AD-Q8_0 | 131.60 GB | 0.00442 | 97.22% | unsloth Q8_0 | 131.61 GB | 0.00443 | 97.23% |

| AD-Q6_K | 120.00 GB | 0.00476 | 97.13% | unsloth UD-Q6_K_XL | 120.99 GB | 0.00482 | 97.08% |

| AD-Q5_K_S | 110.57 GB | 0.00603 | 96.76% | unsloth UD-Q5_K_XL | 110.70 GB | 0.00628 | 96.66% |

| AD-Q4_K_S | 86.90 GB | 0.00816 | 96.23% | unsloth UD-Q4_K_XL | 86.91 GB | 0.00937 | 95.98% |

| AD-IQ4_NL | 74.83 GB | 0.01327 | 95.24% | lmstudio-community NVFP4 | 75.24 GB | 0.02564 | 93.16% |

| AD-IQ4_NL | 74.83 GB | 0.01327 | 95.24% | unsloth NVFP4 | 77.69 GB | 0.02530 | 93.17% |

| AD-IQ3_M | 69.97 GB | 0.02366 | 93.75% | lmstudio-community Q4_K_M | 70.10 GB | 0.04527 | 90.97% |

| AD-IQ3_XXS | 65.34 GB | 0.03523 | 92.43% | unsloth UD-Q3_K_XL | 65.76 GB | 0.04433 | 91.61% |

| AD-IQ2_M | 59.32 GB | 0.06429 | 89.75% | unsloth UD-IQ3_XXS | 59.76 GB | 0.06702 | 89.83% |

| AD-Q2_0_XL | 57.43 GB | 0.06566 | 90.02% | unsloth UD-Q2_K_XL | 57.79 GB | 0.07666 | 89.09% |

| AD-Q2_0_L | 55.84 GB | 0.07219 | 89.44% | unsloth UD-IQ2_M | 55.88 GB | 0.08837 | 88.27% |

| AD-Q2_0_M | 51.02 GB | 0.10652 | 86.97% | unsloth UD-IQ2_XXS | 51.12 GB | 0.11707 | 86.50% |

| AD-Q2_0_S | 45.72 GB | 0.14704 | 84.86% | unsloth UD-IQ1_M | 46.14 GB | 0.15571 | 84.39% |

The Q8_0 pair is a tie: both are plain q8_0, as they should be. AD-IQ2_M

drifts less than UD-IQ3_XXS on average but matches the original's top token

slightly less often.

Our files were measured on 8x A100-SXM4-80GB, theirs on 8x A100-SXM4-40GB, same

llama.cpp commit and CUDA build; the Q8_0 pair, which should be equal, comes

out within 0.2% of each other across the two machines.

What we found while building these

• Every expert in this model is 2688 wide, which is not a multiple of 256. The

expert down projections can therefore only take llama.cpp's 32-wide block

types (q4_0, q5_0, q8_0, iq4_nl), or q2_0. Our files write those types

directly instead of letting llama-quantize swap them in silently.

• The blocks that get extra precision (76, 79, 81, 83, 85, 87) were picked from

the importance matrix statistics, not by position.

• Below 60 GB the down projections are the problem: iq4_nl (4.5 bits) is the

lowest calibrated type they take, which leaves too few bytes for the up

projections. Unsloth's files solve it with q2_0 on the down projections of some

blocks. We do the same with our own q2_0 quantizer (above), which is where the

gain at these sizes comes from.

• Which blocks take q2_0 matters as much as the quantizer. Our importance-matrix

ranking grows with depth on this model and picked the wrong blocks: with the

same quantizer it measured 8 to 18% worse than the block sets in Unsloth's

files, which keep blocks 19 to 26 and 72 to 87 at iq4_nl. The AD-Q2_0_* files

use those block sets; the quantizer is ours.

• On this model the expert up projections matter more than the down projections.

Our first 87 GB file kept the down projections a step above the up ones and

drifted 0.0104; AD-Q4_K_S puts q5_k on every up projection, takes the down

projections' layout from UD-Q4_K_XL (iq4_nl on blocks 28 to 43, q5_0

elsewhere) and spends what is left on the Mamba, attention, latent and shared

expert tensors at q8_0: 0.0082.

The calibration data

Our importance matrix comes from a corpus we built for this model and

published: AtomicChat/calib-corpora,

recipe nemotron-omni-moe. It is 4,589,438 tokens across 2,820 documents:

| Part | Share |

|---|---:|

| agentic and tool use | 24.0% |

| multilingual | 19.6% |

| code | 17.4% |

| reasoning | 13.1% |

| long context | 11.1% |

| vocabulary sweep | 5.9% |

| structured data | 4.4% |

| graphics | 4.4% |

Every conversation is rendered through this model's own chat template, so

the chat markup appears as the single tokens the model actually sees.

> [!IMPORTANT]

> If you build your own importance matrix from this corpus, pass

> --parse-special to llama-imatrix. Without it the chat markup is read as

> ordinary punctuation.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized.

Perplexity on the held-out set is 6.3673 plus or minus 0.03710.

Held-out text: eval_neutral from our calibration dataset, never used for

calibration. 95 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the

reference, plus top-1 agreement.

Hardware: 8x A100-SXM4-80GB.

The raw logits of the reference are published in the metrics repo, split into

parts because of the file size limit. With them you can measure your own build

against exactly the same point we did:

cat base-neutral.kld.*.part > base-neutral.kld

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a

quantized copy, split across 8 workers and merged. Splitting by chunk

range gives the same result as one long run, because the statistic is a sum:

llama-imatrix -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf -f calib_train.txt -o shard-0.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special --from-chunk 0 --chunks 1121

llama-imatrix -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf --in-file shard-0.gguf,shard-1.gguf,... \
  -o imatrix.gguf

Then, for AD-IQ3_M:

llama-quantize \
  --imatrix imatrix.gguf \
  --tensor-type '^blk\.88\.ffn_down_exps\.weight$=q5_1' \
  --tensor-type '^blk\.88\.=q5_k' \
  --tensor-type '^token_embd\.weight$=q8_0' \
  --tensor-type '^output\.weight$=q8_0' \
  --tensor-type '^blk\.(76|79|81|83|85|87)\.ffn_down_exps\.weight$=q5_0' \
  --tensor-type '^blk\.(76|79|81|83|85|87)\.ffn_up_exps\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_down_exps\.weight$=iq4_nl' \
  --tensor-type '^blk\.\d+\.ffn_up_exps\.weight$=iq3_s' \
  --tensor-type '^blk\.\d+\.ffn_latent_(down|up)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ffn_(down|up)_shexp\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.attn_(q|k|v|qkv|output)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ssm_(in|out)\.weight$=q8_0' \
  --override-kv general.file_type=int:26 \
  NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-AD-IQ3_M.gguf IQ3_M

The rules for every other file are in ladder/ in the metrics repo, next to

its quantization log. The first rule that matches a tensor wins.

Model details

88 layers (40 Mamba-2, 40 mixture-of-experts with 512 experts, 8 attention),

hidden size 4096, vocabulary 131,072. One multi token prediction head. NVIDIA's

config sets a maximum context of 1,048,576 tokens; we measured at 4096 and did not

test long context.

Needs a recent llama.cpp with nemotron_h_moe architecture support. Built and tested with llama.cpp b11668 (cd0a5b6a1).