---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

library_name: gguf

pipeline_tag: image-text-to-text

tags:

- gguf

- llama.cpp

- nemotron

- mixture-of-experts

- expert-pruning

- reap

- 16gb

- tool-calling

- code

---

Nemotron 3.5 Bolt 74B-A9B on a 16 GB GPU (expert-pruned GGUF)

A coding-focused build: experts were chosen on a mix that was 70% code, coding-agent traces and

function calls, and only coding and tool calls were benchmarked.

Run Nemotron 3.5 Bolt on a 16 GB card. NVIDIA's compact Bolt build is sized for 24 GB. These two

GGUFs remove routed experts from it so the whole model sits on a 16 GB or a 12 GB GPU, with

room left for a long context window, and with no re-quantization: every expert that remains is byte for byte

NVIDIA's quantization-aware-trained 2-bit weights.

| | NVIDIA Bolt 24GB | This repo: 16 GB | This repo: 12 GB |

|---|---|---|---|

| File | 20.5 GB | 12.5 GB | 8.4 GB |

| GPU memory allocated at 32K context (measured at server start) | 20.7 GB | 12.7 GB | 8.6 GB |

| GPU memory allocated at 128K context (measured at server start) | 21.6 GB | 13.5 GB | 9.5 GB |

| Peak GPU memory during a 124K-token request (measured) | 21.6 GB | 13.6 GB | 9.6 GB |

| HumanEval+ (pass@1) | 0.890 | 0.835 (94% kept) | 0.579 (65% kept) |

| MBPP+ (pass@1) | 0.743 | 0.688 (93% kept) | 0.571 (77% kept) |

| Tool calls, BFCL simple (valid / correct) | 100% / 94.3% | 99.5% / 90.8% | 99.3% / 89.0% |

| Tool calls, BFCL multiple (valid / correct) | 99.0% / 92.0% | 99.0% / 93.0% | 97.5% / 89.0% |

| Decode / prefill speed (tok/s, DGX Spark) | 40.1 / 1,145 | 41.4 / 1,398 | 42.3 / 1,567 |

Memory figures are for text; the vision projector adds about 1.7 GB (see Run it). Sizes are in GB

the way GPUs count them (1 GB = 1,024 MB), so they compare directly with your card's 16 GB. The

Hugging Face file list counts in decimal, so it shows the 16 GB build's file as 13.4 GB.

The 16 GB build keeps 93 to 94% of the 24GB build's coding scores and its tool calling, and a

real 124K-token request peaked at 13.6 GB of GPU memory, inside a 16 GB card. Long-context quality

was not benchmarked. The 12 GB build is for cards that cannot fit the 16 GB one: a 124K-token

request peaked at 9.6 GB and it still calls tools reliably, but it loses about a third of its HumanEval+ score.

Files

| File | Use |

|---|---|

| Nemotron-3.5-Bolt-74B-A9B-16GB-REAP50-IQ2_XXS.gguf | 16 GB GPUs |

| Nemotron-3.5-Bolt-74B-A9B-12GB-REAP25-IQ2_XXS.gguf | 12 GB GPUs. Aggressive: keeps 65% of HumanEval+ and 77% of MBPP+ |

| mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf | Vision projector, NVIDIA's file unchanged. Add --mmproj <file> for image input |

| selections/*.json | The exact experts kept in every layer |

| eval/ | Scores with paired CIs, per-problem results, tool-call outputs, and provenance (source revision and hash, llama.cpp commit, calibration size, expert-score stability, tensor verification) |

74B-A9B names the parent model. REAP50/REAP25 mean 50%/25% of each layer's routed experts were kept.

Run it

Any llama.cpp build that runs NVIDIA's official Bolt 24GB GGUF should run these: same

architecture, same format, fewer experts. Evaluated with llama.cpp at commit 547208d, plus a small

local patch that returns malformed chat output as plain text instead of an HTTP error; it never

triggered in any run behind this card.

llama-server -m Nemotron-3.5-Bolt-74B-A9B-16GB-REAP50-IQ2_XXS.gguf -ngl 99 -c 32768 \
  --parallel 1 --jinja --reasoning off --reasoning-format deepseek

These are the settings the memory and long-context numbers on this card were measured with: one slot,

whole model on the GPU, reasoning off. The code and tool-call benchmarks ran with four parallel slots

and the same reasoning settings; speed comes from llama-bench. Leave out --reasoning off to let the model think first (not evaluated).

Raise -c to 131072 for 128K context; llama.cpp then allocates 13.5 GB for the 16 GB build.

LM Studio and Ollama have not been tested yet.

For image input add --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf. The projector adds

about 1.7 GB of GPU memory: the 16 GB build then allocates 14.3 GB at 32K context and 15.2 GB

at 128K, which leaves less than 1 GB free on a 16 GB card: too tight if the card also drives a display. For long context with images, add --no-mmproj-offload

to keep the projector on the CPU: GPU memory stayed at 13.6 GB in our test, but each image took

about 40 s to encode on the DGX Spark's CPU. The 12 GB build with the projector allocates 10.3 GB

at 32K and 11.1 GB at 128K.

How it was made

Nemotron 3.5 Bolt routes each token to 9 of 240 to 496 experts per layer. Most of the 24GB

file is those experts. We measured how much each expert contributes on 3.1 million tokens of

calibration text (2,047 conversations: code, coding-agent trajectories, function calling, and

general chat), scored experts with REAP (router weight times expert output size, averaged over

the tokens routed to it), and kept the top half (16 GB build) or top quarter (12 GB build) of

every layer. Layers keep the same fraction, so NVIDIA's own per-layer expert counts are scaled,

not flattened.

The pruning is a byte copy. Kept experts, the router rows for them, and every non-expert tensor

are identical to NVIDIA's file; this was checked tensor by tensor for every build. Expert scores

were stable across the calibration set: the expert rankings from its two halves correlate at a

median of 0.99 across layers (0.95 in the least stable layer). REAP was chosen over routing mass (router weight alone) by

evaluation: routing mass scored lower on both builds.

Method background: Half the Experts, All the Code, accepted at

the COLM 2026 Workshop on Efficient Reasoning.

Evaluation details

• Code: EvalPlus HumanEval+ (164 problems) and MBPP+ (378), greedy decoding, reasoning off,

served by llama-server. Against Bolt 24GB, the 16 GB build's MBPP+ drop is -5.6 points (95% CI

-9.3 to -1.9) and its HumanEval+ drop is -5.5 points (95% CI -11.0 to 0.0, not significant at

p < 0.05). Paired bootstrap CIs and McNemar tests are in eval/summary.json.

• Tool calls: BFCL v3

simple (400) and multiple (200) through llama-server's OpenAI tools interface. "Valid" means one well-formed call to a provided function; "correct" uses a

simplified version of BFCL's argument checker, so compare these numbers with each other, not

with the official leaderboard.

• Degenerate outputs: 1 of 542 code generations for each build.
• Long context: one passphrase-retrieval check at 124K tokens (passphrase placed 60% of the way in)

passed for both builds; it is a smoke test, not a long-context benchmark.

• Memory: GPU memory of the llama-server process (nvidia-smi), model fully offloaded, one

sequence, measured on an NVIDIA DGX Spark. llama.cpp's own breakdown for the 16 GB build at 32K

context: 12,207 MB weights + 420 MB KV and state + 148 MB compute. These are the allocations

llama.cpp makes at startup (it reserves the full KV cache and compute buffers up front), read once

the server was ready and before any request. Peak memory during one 123,598-token request, sampled every

0.25 s, was 22,165 MB for Bolt 24GB, 13,925 MB for the 16 GB build and 9,806 MB for the 12 GB build,

each 90 MB above its startup allocation. Memory on an actual 12 or 16 GB discrete GPU was not measured. On a desktop GPU that also drives a display,

leave some headroom.

• Speed: llama-bench, 512-token prefill and 128-token decode, DGX Spark. Decode speed depends on

the experts active per token, which pruning does not change; expect your own GPU's numbers.

Limitations

• Calibration is code-heavy. Coding (HumanEval+, MBPP+) and single tool calls (BFCL simple and

multiple) were measured. Multi-step agent workflows, general knowledge, multilingual use,

long-context quality and long-form reasoning were not, and may degrade more.

• Evaluated with reasoning off. Reasoning-on behavior was not measured.
• Vision: smoke-tested only. On six generated images (reading text, counting, colour, a bar chart,

code, a sign), both builds answered 6/6, the same as Bolt 24GB; vision quality was not benchmarked.

License and attribution

Unofficial community derivative of NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB;

not affiliated with or endorsed by NVIDIA. Distributed under the

OpenMDW License Agreement, version 1.1, the license of the

original model; the full text is in LICENSE. All metadata from NVIDIA's GGUF is retained in

these files.

Pruned with moep at commit 400603b.