---

base_model: spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara

base_model_relation: quantized

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

library_name: llama.cpp

pipeline_tag: image-text-to-text

quantized_by: spiritfather

tags:

• gguf
• imatrix
• creative-writing
• heretic
• ara
• uncensored
• abliterated
• mtp
• vision
• llama.cpp

---

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara — GGUF (imatrix (i1) quants)

<p align="center">

🧬 Source model: <a href="https://huggingface.co/spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara" target="_blank">spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara</a> · base: <a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16" target="_blank">nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16</a>

<br>

📦 Static quants: <a href="https://huggingface.co/spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-GGUF" target="_blank">spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-GGUF</a>

<br>

🛠️ Runs on <a href="https://github.com/ggml-org/llama.cpp" target="_blank">llama.cpp</a> (master)

<br>

📊 Scored on <a href="https://caliperbench.com" target="_blank">CaliperBench</a>

<br>

💬 <a href="https://discord.gg/rhE9zdKyE4" target="_blank">Discord</a>

</p>

> [!Note]

> Credit for the base model goes to NVIDIA — NVIDIA-Nemotron-3.5-Super-VL-120B-A12B, a 120B total / 12B active hybrid Mamba-Transformer latent-MoE with a RADIO vision tower and an MTP head. Credit for the method goes to p-e-w and the Heretic contributors (Arbitrary-Rank Ablation, Heretic 2.0). We ran the search and built the checkpoint (spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara); these are its quantizations.

> [!Tip]

> Benchmarked on CaliperBench — a creative-writing benchmark scoring prose craft, roleplay and willingness rather than general intelligence. See this model's scores: caliperbench.com

> [!Important]

> MTP included. NVIDIA's multi-token-prediction head ships as mtp-*.gguf and speeds up decoding when loaded as the draft model (--spec-type draft-mtp; about 1.16x on our Q8_0 test). Built and tested with llama.cpp master; older builds may not load the vision projector or the MTP head.

> Vision: this is a VL model. llama.cpp keeps the vision tower OUT of the main GGUF, so multimodal use needs mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.gguf alongside your chosen quant (llama-server -m <quant>.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.gguf). Text-only use needs just the quant.

> imatrix (importance matrix) computed over bartowski calibration_datav3 on this model's own bf16. Weighted quants spend their bits where the calibration text says the model is sensitive; they are usually preferable to a static quant of the same size below Q5.

This is a decensored version of nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, made using Heretic v2.0.0.dev0

Arbitrary-Rank Ablation (ARA) parameters

| Parameter | Value |

| :-------- | :---: |

| start_layer_index | 41 |

| end_layer_index | 54 |

| preserve_good_behavior_weight | 0.7358 |

| steer_bad_behavior_weight | 0.1777 |

| overcorrect_relative_weight | 0.9892 |

| neighbor_count | 2 |

Performance

| Metric | This model | Original model (nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) |

| :----- | :--------: | :---------------------------: |

| Refusals | 6/100 | 100/100 |

| KL divergence | 0.1092 | 0 (by definition) |

Heretic's own evaluation: refusals are Heretic's keyword rule over 100 held-out prompts from mlabonne/harmful_behaviors; KL divergence is first-token KL against the original model over 100 prompts from mlabonne/harmless_alpaca. Thinking off. It is not a safety evaluation and says nothing about what the model will or will not produce.

Provided quants

| Quant | Size | PPL | KLD vs BF16 | Same top token | Notes |

|:--|--:|--:|--:|--:|:--|

| i1-Q4_K_M | 86.1 GB | 4.9413 | 0.0308 | 93.0% | fast, recommended |

| i1-IQ4_XS | 66.3 GB | 5.0161 | 0.0405 | 92.3% | best size/quality at 4 bits |

| i1-IQ3_M | 60.8 GB | 5.1433 | 0.0983 | 87.8% | |

| i1-IQ3_XXS | 56.8 GB | 5.5039 | 0.1646 | 84.6% | lower quality |

| i1-IQ2_M | 52.9 GB | 6.6182 | 0.3646 | 75.8% | lowest quality, for tight memory |

PPL and KLD: wikitext-2 test set, 100 chunks of 512 tokens, every quant scored against the logits of our BF16 GGUF. Same top token is how often the quant's most likely next token matches BF16's. Quants over 45 GB are split with llama-gguf-split into a folder of shards; point llama.cpp at the first shard (or use -hf) and it loads the rest.

Sidecars (top level, shared by every quant): mmproj-.gguf is the vision projector (BF16, Q8_0) and mtp-.gguf is NVIDIA's multi-token-prediction head as a separate draft model (BF16, Q8_0, Q4_0). With -hf, llama.cpp downloads the mmproj automatically and the MTP head when --spec-type draft-mtp is set, each at the precision closest to the quant.

Loading (any current llama.cpp)

llama-server -hf spiritfather/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-i1-GGUF:IQ4_XS --spec-type draft-mtp -ngl 99 -fa on -c 32768 --jinja

# local files: the quant (first shard if split) + sidecars
llama-server -m <QUANT>/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara.<QUANT>-00001-of-0000N.gguf --mmproj mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.gguf \
  -md mtp-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-heretic-ara-Q8_0.gguf --spec-type draft-mtp -ngl 99 -fa on -c 32768 --jinja

Thinking: on by default in the template; pass chat_template_kwargs: {"enable_thinking": false} per request to turn it off.

Quant recipe

Cut directly from our BF16 GGUF with an importance matrix computed on that same BF16 (bartowski calibration_datav3); the imatrix file is in this repo. Sidecars are the same files as in the static repo.

License

OpenMDW 1.1, inherited from nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.