---

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

base_model_relation: quantized

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

pipeline_tag: text-generation

library_name: gguf

tags:

• gguf
• llama.cpp
• nemotron_h
• moe
• quantization
• imatrix

---

Nemotron-3.5-Lightning-30B-A3B, ShimQuant tuned, 11.77 GiB

> **This file will not load in stock llama.cpp, LM Studio, Ollama, or any other

> unpatched runtime.** It needs the ShimQuant patch.

> The failure is immediate and explicit, not silent corruption:

>

> ```

> check_tensor_dims: tensor 'blk.0.ssm_in.weight' has wrong shape;

> expected 2688, 10304, got 2816, 10304

> llama_model_load_from_file_impl: failed to load model

> ```

>

> Read Requirements before downloading 11.77 GiB.

A 3.07 bits-per-weight build of Nemotron-3.5-Lightning that runs on a 16 GB consumer card at

262,144 tokens of context. The stock quantizer cannot produce a file this size for this model

at any label.

11.77 GiB = 12.63 GB. Both units appear below because comparable published work uses

decimal GB and mixing them makes files look smaller than they are.

About the name. The file says IQ3_XXS because IQ3_XXS is nominally 3.0625 bpw and this file

measures 3.069, which is closer than any other standard label gets. The tensors are not all

IQ3_XXS. The recipe is a Q6_K base with iq2_xxs on gate and up, iq2_s on down, and q8_0 pinned on

blk.52, and GGUF has no field that can say that. general.file_type is set to IQ3_XXS to match the name. That

declares the bit budget, not the tensor types, which is the closest honest thing the format allows.

llama-quantize leaves it at the base type you passed, so it read Q6_K until I corrected it by hand.

Why this exists

Nemotron-3.5-Lightning has n_embd = 2688 and expert widths of 1856 and 3712. None divide by

256, so no k-quant or i-quant can legally apply to about 99% of its parameters.

llama-quantize substitutes a 32-block type instead and keeps the requested filename, so

every published low-bit quant of this model measures ~4.70 bpw regardless of its name. The

four IQ2 rungs in the popular repos are the same file under four labels. Census across 25

repos and 443 quants: ggufaudit.

ShimQuant shims each affected tensor row out to the next multiple of 256, so the low-bit types

actually apply. The shim is zero-padding written at quantize time and sliced back off the

activations at inference. The expert banks here shim 1856 to 2048, which is 9.4% overhead.

That small distance to the next multiple is why this model is a good candidate and others are

not.

Prior report of the underlying mechanism on this model:

worthant, 2026-08-12.

Divergence from the Q8_0 reference

Mean per-token KL divergence in nats against one shared answer key: Q8_0 logits dumped once

from the stock build, teacher-forced over a fixed corpus, identical for every row.

| build | measured bpw | size | KLD vs Q8_0 |

|---|---:|---:|---:|

| stock IQ3_XXS | 4.70 | 18.01 GiB | 0.0398 |

| stock IQ2_M | 4.70 | 18.00 GiB | 0.2722 |

| shimmed IQ3_XXS | 3.58 | 13.71 GiB | 0.2541 |

| shimmed IQ2_M | 3.13 | 12.01 GiB | 0.5229 |

| this file (shimmed, tuned) | 3.07 | 11.77 GiB | 0.1230 |

Note the two middle rows. Shimming on its own is worse than stock, and by a wide margin.

It does not improve anything; all it does is make the low-bit types legal on these tensor widths.

The tuned recipe is what turns that access into a usable file, and without it this technique looks

like a bad trade.

Read that table honestly. Against stock IQ2_M this file is 6.2 GiB smaller and less

divergent, which is a strict improvement. Against stock IQ3_XXS it is 6.2 GiB smaller and

three times more divergent, which is a trade. This file does not dominate the published

ladder and the claim here is narrower: below about 18 GiB the stock quantizer produces nothing

at all for this model, and this is a usable file in that gap.

Capability

HumanEval, greedy decoding, executed tests, 6000-token cap, 164 problems.

| | pass@1 | hit token cap | accuracy when code was emitted |

|---|---:|---:|---:|

| this file | 91.5% | 5.5% | 97.4% |

Against other published builds of this model

Same harness, same 164 problems, same greedy settings, same token budget, same machine. Sizes

in decimal GB. Comparison files are from

AtomicChat,

whose published divergence figures are measured against a different corpus and are therefore

not comparable to the table above. This is why they were re-scored here rather than quoted.

| build | maker | size | pass@1 | hit token cap |

|---|---|---:|---:|---:|

| AD-Q2_0 | AtomicChat | 11.39 GB | 0.0% | 59.8% |

| this file | ShimQuant | 12.63 GB | 91.5% | 5.5% |

| AD-Q2_0_L | AtomicChat | 15.52 GB | 37.8% | 39.6% |

| AD-IQ4_NL | AtomicChat | 19.65 GB | 91.5% | 4.9% |

AD-Q2_0 produced no parseable function in 164 attempts, which matches its maker's own

assessment of it as unusable. AD-IQ4_NL is that repo's recommended build and it ties this

file exactly, at 7.02 GB larger. The AtomicChat repo states plainly that **"there is currently

no good 16 GB option for this model, from anyone."** That was accurate when written.

Measured on a 16 GB card

RTX 5080, 16,303 MiB, full offload, patched build a41d55cb8.

| | value |

|---|---|

| maximum context loaded | 262,144 |

| peak VRAM at that context | 13,458 MiB of 16,303 |

| peak VRAM at 65,536 context | 13,171 MiB |

Context is nearly free on this model because most of its layers are SSM rather than attention,

so there is no KV cache growing with sequence length. Going from 65,536 to 262,144 tokens costs

287 MiB. 262,144 was the largest value tested, not a measured ceiling.

Throughput

A single generation number is not meaningful for this file, because the two things that move

it most are prompt depth and whether MTP is on. All of the following is one card, one file,

-c 16384, greedy, full offload.

| prompt | workload | no MTP | with MTP | gain |

|---|---|---|---|---|

| 906 tok | code | 139.8 tok/s | 148.1 tok/s | 1.06x |

| 12,346 tok | prose | 73.6 tok/s | 144.4 tok/s | 1.96x |

| 14,306 tok | code | 64.9 tok/s | 109.4 tok/s | 1.69x |

Prefill runs 397 to 745 tok/s without MTP and 582 to 667 with it, rising with prompt length in

both cases.

The gain grows with prompt depth. MTP buys little on a short prompt and close to a doubling at

12,000 tokens and up, so an agent working in a real codebase gets far more from it than a chat

turn does. A separate sweep at shorter prompts and a 120-token budget measured 1.2x fairly

flatly from 231 to 906 tokens, so the shallow figure depends on how much you generate as well

as on prompt length; treat 1.06x and 1.2x as the same weak result rather than a discrepancy.

Every number above is wall-clock verified. That matters more than it sounds: llama-server's

reported timings.predicted_per_second does not account for draft-side work, and on a

different model we measured it reporting 38.6 tok/s during a request that took 537 seconds.

It agrees with wall clock for this file, but do not benchmark any speculative config from that

field alone.

To turn MTP on, add --spec-type draft-mtp --spec-draft-n-max 3. No draft model argument is

needed; the head is blk.52 inside this file, which is why it is pinned to q8_0 in the recipe

below rather than pruned out.

On quality: speculative decoding is supposed to be output-preserving, since the draft only

proposes and the target verifies. Scored on the full 164-problem harness with and without the

flag, 9 of 164 outcomes changed, 6 down and 3 up. The measurement noise floor on this harness

is exactly 0.0000 over six identical runs, so those 9 flips are real differences caused by MTP

rather than variance. Exact McNemar on the paired outcomes gives p = 0.51, so the net direction

is not established, and conditional accuracy is unchanged at 0.9735 against 0.9740. Four of the

six losses are problems that hit the token cap only with MTP, so what shifts is which problems

run long, not whether the model writes correct code.

That is a weaker claim than "free" and a weaker claim than "costly", and it is the one the

evidence supports.

Tool calling

Agentic use needs more than pass@1. A coding worker reads files, runs commands and searches,

which means emitting a well-formed tool call with the right name and the right arguments. That

is a stricter target than a function body, because a wrong token inside a JSON argument is a

hard failure with no partial credit.

170 cases: 160 across four tools (read_file, run_command, search_code, list_dir), plus

10 questions that need no tool at all.

| | |

|---|---|

| emitted a well-formed tool call | 160/160 |

| chose the correct tool | 159/160 |

| correct tool and correct arguments | 159/160, 0.994 |

| did not call a tool when none was needed | 10/10 |

The single failure routes "What does Makefile contain?" to list_dir instead of read_file.

The restraint row matters as much as the accuracy row. Over-calling is the usual failure mode

of small quantized models, and a benchmark that only rewards calling would score a model that

calls constantly as perfect. Ten cases is a small sample and this row should be read as

indicative rather than precise.

For context, the same harness run across a full qwen3-30b-a3b ladder moves from 1.000 at Q8_0

to 0.981 at IQ2_M, with no rung significantly different from its own Q8_0 reference under exact

McNemar. Tool calling appears to be considerably more robust to quantization than code

generation is, which was the opposite of what we expected going in.

Requires --jinja. Without it llama-server ignores the tools array and replies in prose.

Requirements

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout e70802a01f03f0ed31a26338a5664796f3824371
git apply /path/to/shimquant.patch      # github.com/JoshBolding/shimquant
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

The commit is pinned because every number here was measured on a build of it. As of

2026-08-28 the patch also applies cleanly to upstream master (50f068fff), verified with

git apply --check, but that build is unmeasured.

./build/bin/llama-server -m Nemotron-3.5-Lightning-30B-A3B-ShimQuant-IQ3_XXS.gguf -ngl 99 -c 32768

Backends

| backend | status |

|---|---|

| CUDA | measured on RTX 3090 and RTX 5080 |

| Metal | builds and runs correctly on Apple M2 Pro; perplexity on a shimmed toy model matches CPU to within 0.002% |

| CPU | works |

| Vulkan, ROCm | untested, no hardware |

Shimming does not change the math

The concern with a graph-level change is that it quietly alters results. It does not. The same

model quantized to Q8_0 twice, once normally and once with shimming forced on for eight tensors

including a 320 to 512 widening, gives identical perplexity:

Q8_0, no shim    PPL = 260.7309 +/- 0.54015
Q8_0, shimmed    PPL = 260.7309 +/- 0.54015

The recipe

Q6_K everywhere except the expert banks, which carry about 93% of the parameters and are the

cheapest class to crush on this model, measured by single-class ablation:

llama-quantize --imatrix nemotron.imatrix \
  --tensor-type "blk.52.=q8_0" \
  --tensor-type "ffn_(gate|up)_exps=iq2_xxs" \
  --tensor-type "ffn_down_exps=iq2_s" \
  BF16.gguf output.gguf Q6_K 24

blk.52 is the MTP block. It is decode-only, gets no imatrix data, and llama-quantize

refuses low-bit types on tensors without importance statistics, so it must be pinned.

Limitations

• Patched build only. Stated three times because it is the real cost of using this.
• Does not beat stock IQ3_XXS on divergence. See the table above. The claim is about a

size range the stock quantizer cannot reach, not about winning everywhere.

• Reasoning is on by default and --reasoning-budget 0 does not disable it on this

architecture. About 5% of HumanEval problems exhaust the token budget inside the thinking

block without emitting an answer. Raise max_tokens if you see empty replies. Doubling the

budget to 12,000 raises pass@1 to 93.9% and cuts the capped rate to 3.0%.

• Evaluated on code and general text. No multilingual, long-context, or agentic evaluation,

despite the context figure above being a load test rather than a quality test.

• One model, one recipe. ShimQuant does not help everywhere. On Qwen3.8-Flash-Next the

same technique produced a worse result than the existing published quants, because 29% of

that model is a per-layer embedding table that shimming cannot touch. Documented in the patch

repo.

Credits

• NVIDIA for the

original model, released under OpenMDW-1.1, which governs

this derivative.

• Unsloth for the BF16 GGUF conversion this was quantized from.
• worthant / AtomicChat for reporting the mechanism on

this model first, and for publishing a measured ladder careful enough to compare against.

• The llama.cpp maintainers, and

PR #3747, which introduced the dimension

fallback this works around. Deliberate behaviour, not a bug.