---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16

base_model_relation: quantized

pipeline_tag: image-text-to-text

library_name: gguf

language: [en, es, fr, de, it, ja]

tags: [gguf, llama.cpp, nemotron, nemotron-3.5, mamba, moe, vision, imatrix]

---

NVIDIA Nemotron 3.5 Bolt 74B-A9B — GGUF, 3- to 8-bit, from the BF16 weights

GGUF quants of nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16

for llama.cpp: a 74B Mamba-2 / Transformer hybrid with LatentMoE (9B active) and a C-RADIO v4-H vision encoder.

NVIDIA's own GGUF is a single 2-bit file trained for that precision (the 24 GB QAD checkpoint). These files are

quantized from the full-precision BF16 weights instead, at 3 to 8 bits, for machines with more memory:

2 files from 28 to 45 GiB, plus the vision projector.

Which file

The routed experts are ~89% of the weights, so every file runs on a GPU of any size by keeping some experts in system RAM.

Rows: total memory you need = VRAM + RAM. Pick the largest file whose RAM column fits your machine.

| file | size GiB | 8 GB card | 12 GB card | 24 GB card | 32 GB card |

|---|---|---|---|---|---|

| Q4_K_M/ (2 parts) | 45.0 | -ngl 67 -ncmoe 88<br>~6.8 GiB VRAM + 40 GiB RAM | -ngl 99 -ncmoe 86<br>~9.3 GiB VRAM + 37 GiB RAM | -ngl 99 -ncmoe 62<br>~21.2 GiB VRAM + 25 GiB RAM | -ngl 99 -ncmoe 46<br>~28.2 GiB VRAM + 18 GiB RAM |

| NVIDIA-Nemotron-3.5-Bolt-74B-A9B-IQ3_XXS.gguf | 28.3 | -ngl 99 -ncmoe 88<br>~6.7 GiB VRAM + 24 GiB RAM | -ngl 99 -ncmoe 80<br>~9.9 GiB VRAM + 20 GiB RAM | -ngl 99 -ncmoe 38<br>~21.6 GiB VRAM + 9 GiB RAM | -ngl 99 -ncmoe 9<br>~28.8 GiB VRAM + 1 GiB RAM |

Quick pick (largest file that fits, keeping 8 GB of RAM for everything else):

| GPU \ system RAM | 32 GB | 64 GB | 96 GB | 128 GB |

|---|---|---|---|---|

| 8 GB | — | Q4_K_M | Q4_K_M | Q4_K_M |

| 12 GB | IQ3_XXS | Q4_K_M | Q4_K_M | Q4_K_M |

| 24 GB | IQ3_XXS | Q4_K_M | Q4_K_M | Q4_K_M |

| 32 GB | Q4_K_M | Q4_K_M | Q4_K_M | Q4_K_M |

Being uploaded: Q8_0, Q6_K, Q5_K_M, IQ4_XS, Q3_K_M (each one after its quality check; this card updates as they land).

Download:

pip install -U huggingface_hub   # provides the `hf` command
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF --include "Q4_K_M/*" --local-dir .   # 45 GiB in 2 parts
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF NVIDIA-Nemotron-3.5-Bolt-74B-A9B-IQ3_XXS.gguf --local-dir .   # 28 GiB
hf download darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf --local-dir .   # vision (BF16 version: mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16.gguf)

Flags are for a recent llama.cpp; with the default --fit on, llama.cpp chooses the offload itself and you can leave

-ngl / -ncmoe out (see "Run"). VRAM figures count weights + 32K context + compute buffers. VRAM numbers are computed from the files (weights + KV + compute estimate); measured values replace them after the B70 runs.

Quality

KL divergence and top-1 token agreement against the Q8_0 file, and perplexity, on wikitext-2 test (32,768 tokens,

2048-token windows). Q8_0 is the reference because the BF16 model (138 GiB) does not fit the test machine; Q8_0 is

normally within noise of BF16.

| file | size GiB | KL vs Q8_0 (mean) | KL 99th pct | same top-1 token | PPL |

|---|---|---|---|---|---|

| Q8_0 (upload follows) | 73.4 | reference | | | 4.5410 |

| Q4_K_M | 45.0 | 0.015014 | 0.117461 | 94.055% | 4.563174 ± 0.076549 |

| IQ3_XXS | 28.3 | pending | pending | pending | pending |

Compared with NVIDIA's 24 GB file

NVIDIA's 24 GB GGUF is 2-bit,

recovered with quantization-aware distillation (QAD): IQ2_XXS routed and shared experts, NVFP4 Mamba-2 in/out projections,

Q8_0 attention, LatentMoE projections, embedding and output. It fits a 24 GB card whole at 32K context; NVIDIA's own

llama.cpp recipes are for 32 GB+ GPUs (64K context, q8_0 KV cache) and 48 GB+ (256K). The files here trade memory for

fidelity to the BF16 weights. Same measurement as above (KL and top-1 vs our Q8_0, wikitext-2 test):

| file | size GiB | KL vs Q8_0 (mean) | KL 99th pct | same top-1 token | PPL | on a 24 GB card | on a 32 GB card |

|---|---|---|---|---|---|---|---|

| NVIDIA 24GB (QAT IQ2_XXS) | 20.5 | 0.169377 | 1.590810 | 84.207% | 5.284233 ± 0.096054 | fits whole | fits whole |

| Q4_K_M (this repo) | 45.0 | 0.015014 | 0.117461 | 94.055% | 4.563174 ± 0.076549 | 21 GiB VRAM + 25 GiB RAM | 28 GiB VRAM + 18 GiB RAM |

| Q4_K_M without imatrix | 45.0 | 0.015134 | 0.118498 | 94.178% | 4.566191 ± 0.076668 | 21 GiB VRAM + 25 GiB RAM | 28 GiB VRAM + 18 GiB RAM |

NVIDIA's file is quantized from its own QAT-trained checkpoint, so its KL against our BF16-based Q8_0 includes that retraining, not only rounding error; PPL compares both on equal terms.

Memory: weights + 32K context + compute buffers (computed; on the B70 NVIDIA's file measured 22.9 GiB with its LiLiCorr drafter at 32K). Speed on the B70: NVIDIA's file 43 tok/s decode (Mesa 26.2); ours in the speed table below.

Speed: one Intel Arc Pro B70 (32 GB)

llama.cpp Vulkan, Mesa 26.2.5, Ryzen 9 5950X, 64 GB system RAM. Flags from the 8 / 12 / 24 GB rows above (the B70 then uses

only that much VRAM). Prompt 512 tokens; generation 128 tokens at empty and 16K context.

| file | card size (flags) | VRAM used | prompt t/s | gen t/s | gen t/s at 16K |

|---|---|---|---|---|---|

| pending | | | | | |

Run

Needs llama.cpp b11669 or newer (October 15, 2026). Bolt's MoE layers have different expert counts (240 to 496), and

llama.cpp loads that only since #30520, which also added the vision

encoder. Older builds stop at load time with a tensor shape error, for text too.

| runtime | loads these files? (checked 2026-10-15) |

|---|---|

| llama.cpp b11669+ | yes: the numbers on this card come from b11669 (620d2225c), Vulkan, on an Arc Pro B70 |

| Ollama | 0.40.5 or newer (bundles llama.cpp b11669; 0.40.4 and older can't load Bolt). ollama run hf.co/darioooooo0o/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF:<quant> works for files stored as one GGUF at the repo root; the Hugging Face registry refuses split (-0000N-of-) files, so for those download the parts and ollama create from them. Not tested by us yet. |

| LM Studio | needs a llama.cpp runtime (extension pack) built on b11669 or newer; not verified by us. LM Studio's community repo offers NVIDIA's file, and these files use the same architecture metadata and standard K / IQ quant types. |

# any GPU (CUDA, Vulkan, SYCL, Metal) or CPU; --fit (default on) picks the offload for your VRAM
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf \
  -c 32768 -fa on --temp 1.0 --top-p 0.95 --jinja

# or set the offload yourself (example: 24 GB card, flags from the table)
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf -ngl 99 -ncmoe 62 -c 32768 -fa on --temp 1.0 --top-p 0.95 --jinja

# CPU only
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf -ngl 0 -c 32768 --temp 1.0 --top-p 0.95 --jinja

# one image from the command line
llama-mtmd-cli -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q8_0.gguf \
  --temp 1.0 --top-p 0.95 --image photo.jpg -p "Describe this image."

Split files (-00001-of-0000N.gguf): point -m at the first part.

Context: NVIDIA's card gives up to 256K tokens (262,144, so -c 262144); its compression recovery trained at 8K, then

128K. Only 8 of the 88 blocks are attention, so the KV cache is small (about 1 GiB at 128K, f16); the 40 Mamba blocks

keep a fixed 160 MiB state per sequence.

Sampling: NVIDIA recommends temperature 1.0, top_p 0.95 (also in its generation_config.json), with reasoning on or off.

Keep --jinja: the reasoning switch and tool calls come from the model's chat template.

Thinking on/off

Reasoning is on by default (the chat template's enable_thinking, default true). Turn it off for the whole server, or

per request:

# server-wide
llama-server -m Q4_K_M/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Q4_K_M-00001-of-00002.gguf ... --jinja --chat-template-kwargs '{"enable_thinking": false}'

# per request (OpenAI-compatible API; Python client: extra_body={"chat_template_kwargs": {...}})
curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Why does binary search take logarithmic time?"}],
  "chat_template_kwargs": {"enable_thinking": false}
}'

Medium reasoning effort: {"enable_thinking": true, "medium_effort": true} (the template appends

{reasoning effort: efficient} to the last user message). Reasoning and the answer share the max_tokens / -n budget;

NVIDIA's examples use 16384. By default the template drops the reasoning of earlier turns from the history

(truncate_history_thinking, default true).

Tools and vision

• Tool calling: pass tools in the OpenAI-style request to llama-server --jinja. The chat template writes calls as

Qwen3-Coder-style XML (<tool_call><function=name><parameter=...>); NVIDIA's vLLM recipe parses them with

--tool-call-parser qwen3_coder. NVIDIA's 24 GB GGUF card

has a complete tool-calling client for llama-server. We have not tested tool calls with these files yet.

• Images: load the projector with --mmproj (Q8_0, or BF16 for full precision) and send images as image_url content

parts, or use llama-mtmd-cli as above. NVIDIA also lists video input (mp4, up to 2 minutes); we have not tried video

with llama.cpp.

• Languages (NVIDIA's card): English, French, Spanish, Italian, German, Japanese, Hindi, Korean, Brazilian Portuguese and

Chinese. NVIDIA's metadata tags the first six; this card uses the same tags.

Intel Arc (B70, B60, B580)

Use the Vulkan backend with Mesa 26.2 or newer: on the B70, Mesa 25.3 decodes this model about 40% slower (measured

with NVIDIA's 24 GB file: 25 vs 43 tok/s). SYCL works too. Optional: NVIDIA's LiLiCorr draft model

(...-24GB-LiLiCorr-GGUF) for

speculative decoding: --spec-type draft-dflash --spec-draft-model <lilicorr.gguf> --spec-draft-ngl 99 --spec-draft-n-max 6

(+27% on code with the 24 GB file, no gain on prose; acceptance with these BF16-based files not measured yet).

How these were made

1. BF16 safetensors to GGUF with convert_hf_to_gguf.py from llama.cpp master plus a fix for this checkpoint

(riceharvest/llama.cpp bolt-convert): Bolt's Puzzle config sits inside the Omni wrapper

(llm_config), its MoE blocks have 240-496 experts each, and its hidden_act describes the MLP. Tensor names and

metadata match NVIDIA's own GGUF (expert_count 496, moe_latent_size 1024).

1. Importance matrix from a Q6_K file quantized without one (its experts fit this machine's RAM; Q8_0 would page

from disk for hours): 152 chunks of 512 tokens of

calibration_datav3 (mixed prose, code, math,

multilingual) + wikitext-2 train + C++/Python source. The wikitext test split used for evaluation is not in it.

1. llama-quantize with per-tensor overrides. The routed experts take the file's type; the small, always-on tensors are

kept higher (NVIDIA's own QAT file also keeps attention and the LatentMoE projections at Q8_0):

| file | routed experts | attention, LatentMoE proj. | Mamba in/out, shared expert | output / embedding |

|---|---|---|---|---|

| Q8_0 | Q8_0 | Q8_0 / Q8_0 | Q8_0 / Q8_0 | Q8_0 / Q8_0 |

| Q6_K | Q6_K | Q8_0 / Q8_0 | Q8_0 / Q8_0 | Q8_0 / Q6_K |

| Q5_K_M | Q5_K_M | Q8_0 / Q8_0 | Q6_K / Q6_K | Q8_0 / Q6_K |

| Q4_K_M | Q4_K_M | Q8_0 / Q8_0 | Q6_K / Q6_K | Q6_K / Q6_K |

| IQ4_XS | IQ4_XS | Q8_0 / Q8_0 | Q6_K / Q6_K | Q6_K / Q6_K |

| Q3_K_M | Q3_K_M | Q8_0 / Q8_0 | Q5_K / Q5_K | Q6_K / Q5_K |

| IQ3_XXS | IQ3_XXS | Q8_0 / Q8_0 | Q4_K / Q4_K | Q6_K / Q4_K |

Scripts: in this repo's scripts/ folder (convert, imatrix, quantize, eval, per-tier bench, size calculator).

Credits and license

• Model: NVIDIA (NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16).

See NVIDIA's model card for intended use, training data, evaluation, bias, safety and privacy notes.

• License: OpenMDW-1.1, same as the original. A copy is in LICENSE.OpenMDW-1.1, and

NVIDIA's notices are kept in NOTICE.md.

• Runtime: llama.cpp (ggml-org). Calibration text: bartowski's calibration_datav3.
• Quantized by @imdariotoo (GitHub riceharvest).