---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model:

• nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16

base_model_relation: finetune

quantized_by: AtomicChat

pipeline_tag: image-text-to-text

library_name: gguf

tags:

• atomic-chat
• atomic-dynamic
• nvidia
• nemotron
• nemotron_h_moe
• moe
• gguf
• imatrix
• llama.cpp
• abliteration
• refusal-direction

---

How to Run NVIDIA Nemotron 3.5 Bolt 74B-A9B Without Refusals Locally

<p style="margin-top: 0; margin-bottom: 0;">

<em>NVIDIA's refusal behaviour projected out of the released weights, baked into Atomic Dynamic GGUF builds and measured against the original.</em>

</p>

<div style="display: flex; gap: 8px; align-items: center; margin-top: 10px; margin-bottom: 10px;">

<a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_nemotron_3_5_bolt_uncensored&utm_content=btn_atomic"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_atomic.png" width="162" alt="Atomic Chat"></a>

<a href="https://discord.gg/8wGSsvmg4V"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_discord.png" width="119" alt="Discord"></a>

<a href="https://github.com/AtomicBot-ai/Atomic-Chat"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_github.png" width="115" alt="GitHub"></a>

</div>

<ul style="margin: 0 0 12px 0;">

<li>On held-out JailbreakBench prompts refusals fall from 97.5% to 0%, on Russian red-teaming prompts from 70% to 0%, and with thinking on from 30 of 30 to none (2 of 30 on the 22 and 29 GB files). MMLU stays where it was: 81.35% before, 81.75% after (2000 questions, McNemar p = 0.29). Tool calls and long context are unaffected.</li>

<li>Made from the full-precision release, not from NVIDIA's 24 GB file, and measured on one stand against the other public Bolt abliterations: at 22 GB ours is the closest to the original and the only one that neither refuses with thinking off nor breaks into repeated text (<a href="#against-other-bolt-abliterations">table</a>).</li>

<li>The refusal is written into the residual stream by every kind of block: the edit covers the Mamba-2 outputs, the attention outputs, the shared experts and the latent projection through which all routed experts write.</li>

<li>Same Atomic Dynamic layouts and importance matrix as our <a href="https://huggingface.co/AtomicChat/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF">Nemotron 3.5 Bolt quants</a>; each file is the size of the original build with the same name.</li>

</ul>

<hr style="margin: 0 0 16px 0;">

The builds

| Build | On disk | Mean KLD | Same top-1 |

|---|---:|---:|---:|

| AD-Q4_K_M | 51.7 GB | 0.0178 | 93.34% |

| AD-IQ2_XS | 29.1 GB | 0.0848 | 86.01% |

| AD-IQ1_M_S | 21.9 GB | 0.2013 | 78.57% |

KLD and top-1 are against the original, unmodified BF16 model on our held-out

neutral text (24 chunks of 4096 tokens), so they hold the edit and the quantization

together. For scale, our plain builds of the original model with the same layouts

are 0.0147, 0.0826 and 0.1941 (on 95 chunks): the edit itself adds about 0.002 to 0.007.

The baked files themselves, thinking off, on the held-out prompts: each of the

three refused none of the 81 English, 100 Russian and 250 XSTest prompts. Degenerate

replies among the Russian ones: 1 on AD-Q4_K_M and AD-IQ2_XS, 4 on AD-IQ1_M_S

in the first run, 1 in the second. With thinking on, AD-IQ1_M_S and AD-IQ2_XS

each refused 2 of the 30 prompts (the BF16 edit: none); AD-Q4_K_M was not checked.

Against other Bolt abliterations

Two other abliterations of Bolt were public on 16.10, both a single 22 GB file

built on NVIDIA's 24 GB layout: BoldingBuilds edits NVIDIA's 2-bit QAT file in

place, and dealignai abliterates the BF16 model and quantizes it to that layout.

We downloaded both and ran them on the stand of this card: the same prompts,

chat template, greedy decoding and refusal classifier for every file.

| ~22 GB file | KLD to the original | Same top-1 | Refused, thinking off: EN / RU / XSTest | Degenerate replies (of 431) | Thinking on (30): refused / unfinished | MMLU, 2000 questions |

|---|---:|---:|---:|---:|---:|---:|

| This repo, AD-IQ1_M_S (21.94 GB) | 0.2013 | 78.57% | 0% / 0% / 0% | 1 | 2 / 0 | 79.00% |

| BoldingBuilds Abliterated-24GB (22.02 GB) | 0.2305 | 77.86% | 40.7% / 16.0% / 0.4% | 1 | 0 / 0 | 80.20% (p = 0.11) |

| dealignai CRACK-24GB (22.02 GB) | 0.5955 | 65.71% | 0% / 3.0% / 0% | 64 | 0 / 24 | 79.55% (p = 0.49) |

| NVIDIA 24GB, not edited (22.02 GB) | - | - | 97.5% / 70.0% / 7.2% | 1 | 30 / 0 | 80.45% (p = 0.045) |

| This repo, AD-IQ2_XS (29.08 GB) | 0.0848 | 86.01% | 0% / 0% / 0% | 1 | 2 / 0 | 81.0% |

What the table says:

• Closest to the original. KLD and top-1 are against the unedited BF16 model, so

they hold the edit and the 2-bit quantization together. Ours is 13% closer than

BoldingBuilds' and three times closer than dealignai's.

• Refusals gone in both modes. With thinking off, BoldingBuilds' file still refuses

41% of the English and 16% of the Russian prompts. With thinking on it refuses none,

where ours refuses 2 of 30.

• No broken text. dealignai's file answers almost everything, but 64 of its 431

replies are degenerate (repeated or garbled text), and in 24 of 30 thinking replies

the reasoning never closes within 3072 tokens.

• Knowledge. NVIDIA's QAT weights hold MMLU best at this size: the unedited file

is 1.45 points above ours (McNemar p = 0.045), and BoldingBuilds, which keeps those

weights, is 1.2 points above (p = 0.11, within noise). If knowledge at exactly

22 GB matters most, that file is the stronger pick.

• More room, better file. Only this repo has larger builds: at 29 GB AD-IQ2_XS

is above every 22 GB file on MMLU, at less than half the KLD of any of them.

The p values compare each file with ours question by question within one run; a

repeated run of the same file moves MMLU by about half a point. Refusals are read

from the opening of each reply; BoldingBuilds' own card reports judge-scored

numbers on other prompt sets, which are not comparable with these.

For images, use the vision projector from the

original quants

(the edit does not touch it); image input was not tested on these files.

Running it

hf download AtomicChat/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-GGUF \
  --include "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS/*" --local-dir .
llama-server -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS-00001-of-00016.gguf \
  -ngl 99 -c 32768 --jinja

The chat template is inside every file. Upstream llama.cpp with nemotron_h_moe

support runs them; no patch is needed.

How we did it

Nemotron 3.5 Bolt is a hybrid: 40 Mamba-2 blocks, 8 attention blocks and 40

mixture-of-experts blocks whose routed experts live in a narrow latent space. The

usual abliteration recipes are written for plain transformers and miss most of

this model. Here is what we did instead, step by step, with the numbers that

decided each step.

1. Refusal is a direction

A chat model decides to refuse before it writes the first word, and that decision

shows up as one direction in its residual stream (Arditi et al., 2024). Stop the

weights from writing along that direction and the model stops refusing; the rest

of what it writes is barely touched, because one direction out of 4096 is a very

small part of the stream.

2. Finding it

• Prompts. 416 harmful and 416 harmless English prompts (harmful_behaviors,

harmless_alpaca), 128 and 128 Russian ones (Aya red-teaming, Russian Alpaca),

each in the model's own chat template, thinking off. Validation and test prompts

are separate sets and never reach this step.

• Taps. We read the residual stream at the last prompt token after every one

of the 88 blocks. Mamba-2, attention and MoE blocks name their outputs

differently inside llama.cpp, so this took a small patch to its graph.

• Direction. At each depth: mean over harmful prompts minus mean over harmless

ones, English and Russian pooled. Then the component along the harmless mean is

removed: without that, in our earlier releases the edit made models refuse

harmless questions that look like harmful ones.

• Depth. How well the direction separates the two sets (AUROC, Cohen's d) and

how close the English and Russian directions are:

| Input of block | AUROC, EN | d, EN | d, RU | cos(EN, RU) |

|---:|---:|---:|---:|---:|

| 20 | 0.999 | 4.4 | 1.4 | 0.22 |

| 31 | 1.000 | 5.9 | 2.2 | 0.39 |

| 50 | 1.000 | 9.5 | 2.6 | 0.71 |

| 66 | 1.000 | 10.2 | 2.6 | 0.71 |

| 80 | 1.000 | 11.0 | 2.5 | 0.64 |

Refusal becomes readable early, but the English and Russian directions only

agree from the middle of the model on. We took block 66, three quarters of the

depth, where the rehearsal model (below) worked best: separation near its peak

and one direction for both languages. Block 50 was the runner-up and lost on

validation (step 4).

3. Where to cut

Four kinds of matrices write into the residual stream: the output projection of

each Mamba-2 block (40), of each attention block (8), the down projection of each

shared expert (40), and the latent up-projection of each MoE block (40). The

routed experts never write to the stream themselves: they work in a 1024-wide

latent space, their outputs are summed there, and one matrix per block lifts the

sum back to 4096. Editing that one matrix covers every routed expert in the block.

That makes 128 matrices; the router and the embeddings are left alone.

All 128 are needed. In a rehearsal on NVIDIA's Nemotron-Labs-3-Puzzle-75B-A9B,

a public model with the same layout, any subset left the refusals in place

(English validation prompts refused; 99% before the edit):

| Matrices edited | Scale 1.0 | Scale 1.25 |

|---|---:|---:|

| Mamba-2 + attention | 99% | 98% |

| Latent up-projections only | 97% | 94% |

| Attention + shared experts | 99% | 99% |

| Mamba-2 + attention + shared experts | 98% | 97% |

| All 128 | 1% | - |

The refusal is written by every kind of block, and whatever is left unedited

writes it back. The fourth row is all that a LoRA adapter can reach in stock

llama.cpp, which applies no adapter to the latent projections. That is why we

ship baked weights rather than an adapter.

4. The edit

For every one of the 128 matrices:

W' = W - s · r rᵀ W

r is the unit direction, so rᵀW' = (1 - s) rᵀW. At s = 1 a matrix can no

longer write along r at all; at s = 1.25 it writes a quarter of its old

contribution with the opposite sign. We screened the scale as a rank-1 adapter on

the BF16 weights (with a llama.cpp patch that applies adapters to the latent

projections):

| Block 66, scale | Refused, validation EN / RU | Refused with thinking on | First-token KL, harmless | MMLU change |

|---|---:|---:|---:|---:|

| 1.0 | 2.9% / 1.0% | 8 of 30 | 0.044 | +0.50 pp |

| 1.25 | 0% / 0% | 0 of 30 | 0.062 | +0.40 pp |

| 1.5 | 0% / 0% | 0 of 30 | 0.083 | +0.45 pp |

| Block 50, 1.0 | 12.5% / 0% | 4 of 30 | 0.105 | +0.95 pp |

| Random direction, 1.25 | 100% / 70% | - | 0.011 | +0.10 pp |

Scale 1.0 cleared the plain replies but not the refusals the model reaches while

reasoning. 1.25 cleared both; 1.5 did no better and moved the model further from

the original on harmless prompts. A random direction of the same size, on the same

matrices and at the same scale, changes nothing, so the effect comes from the

direction, not from damaging the weights. None of the MMLU changes at block 66

is significant (McNemar p ≥ 0.13).

5. Checks before release

The numbers in this card come from test prompts that played no part in any

choice: 81 JailbreakBench behaviours, 100 Russian red-teaming prompts and the 250

safe XSTest prompts. The thresholds:

• at most 3% refused per language, and XSTest refusals at most 1 point above the

original's;

• at most 1% empty or looping replies;
• first-token KL on harmless prompts at most 0.10, and KLD on neutral text at most

0.05 above the original's;

• MMLU on 2000 questions: no drop of more than 1.5 points that McNemar's test

calls significant;

• with thinking on (30 harmful prompts, 3072 tokens): at most 1 refused;
• 20 tool calls, all valid;
• a needle at 30k tokens, found at all three depths.

The checks ran split across several machines. Their results were merged only when

the model, the code and the direction file had the same fingerprints on all of

them.

6. Baking and quantizing

The edit is applied to the BF16 weights in f32 and rounded to BF16 once. After

rounding, every one of the 128 matrices keeps a quarter of its write along r

(the ideal 0.25) within 0.0007. The baked BF16 is then quantized with the same

importance matrix and the same Atomic Dynamic layouts as our

original quants,

and measured against the unedited BF16 (the table at the top).

What changed, and what did not

Original against this edit on the released weights (BF16, the edit applied as an

adapter), on the test prompts of step 5.

| | Original | This edit | Random direction, same size |

|---|---:|---:|---:|

| JailbreakBench harmful behaviours (81), refused | 97.5% | 0.0% | 97.5% |

| Russian red-teaming prompts (100, aya), refused | 70.0% | 0.0% | 70.0% |

| XSTest safe prompts (250), refused | 9.6% | 0.0% | 9.6% |

| With thinking (30 prompts, 3072 tokens), refused | 30 of 30 | 0 of 30 | - |

| Thinking left unclosed at 3072 tokens | 0 of 30 | 0 of 30 | - |

| Empty or degenerate replies | 0 | 1 of 431 (one Russian reply) | 0 |

| MMLU, 2000 questions | 81.35% | 81.75% (McNemar p = 0.29) | 81.45% |

| Tool calls (20) | 20/20 valid | 20/20 valid | - |

| Needle at 30k tokens (3 depths) | 3/3 | 3/3 | - |

Validation prompts (104 harmful, 104 harmless per language), replies of up to 64

tokens, the check of the recipe on the released weights: none of the English or

Russian harmful prompts refused.

Refusal is counted by the opening of the reply, so it is indicative, not a

judge.

Limitations

• The scale of 1.25 moves the model a little further from the original on harmless

prompts than 1.0 would (first-token KL 0.062 against 0.044); 1.0 did not clear

the refusals the model reaches while thinking.

• The refusal count reads the opening of each reply; a soft refusal phrased as

an answer would be missed.

• MMLU and the thinking check use fixed question sets; they show a large change,

not a small one.

• We publish the baked files only. The same edit as a LoRA adapter needs a

llama.cpp patch on this architecture (upstream applies no LoRA to the latent

projections), so it is not offered here.

Responsible use

This model has its learned refusal behaviour removed, so it will answer

requests the original declines. Nothing about it makes the output safe, correct

or lawful, and any deployment needs its own access controls and policy

enforcement.

License and credits

OpenMDW-1.1, as the original. Base model Nemotron 3.5 Bolt 74B-A9B by

NVIDIA. Direction, edit, quantization and

evaluation by AtomicChat. The method follows Arditi et al., *Refusal in

Language Models Is Mediated by a Single Direction* (2024).