---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model:

• nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

language:

• en
• es
• fr
• de
• it
• ja

tags:

• gguf
• moe
• mamba
• hybrid
• agentic-coding
• mtp
• speculative-decoding
• imatrix

---

NVIDIA Nemotron 3.5 Lightning 30B A3B — MTP GGUFs

Nemotron-3.5-Lightning carries a native MTP draft head (multi-token

prediction, num_nextn_predict_layers: 1) — a full attention + MoE draft

block co-trained with the trunk. It is embedded in-model at blk.52

(blk.52.nextn.* is present in every file here), so speculative decoding

needs no graft and no extra model.

Quantized from the official llama.cpp-team bf16 master

(ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF), whose weights were

verified byte-identical to NVIDIA's safetensors before building.

---

Recipes

Eight tiers in two families, following the KAT-Coder-V2.5-Dev layout:

• Unsloth Dynamic 2.0 series — UD-IQ4_XS, UD-Q4_K_XL, UD-Q5_K_S.

Every non-expert tensor is held at Q8_0; the routed experts carry the tier's

bit width.

• APEX series — APEX-I-Balanced, APEX-I-Quality, APEX-I-Compact,

APEX-I-Mini, APEX-I-Compact-v2D-lite. Per-layer precision: the first and

last layers are protected, middle layers compressed, shared experts kept

high. The v2D-lite variant adds extra headroom on attention keys/values and

the output projection.

One thing to know about this model: its dimensions (2688 / 1856 / 3712)

are not multiples of 256, so the K-family and IQ-family quant types physically

cannot quantize most of the weights — llama.cpp silently substitutes block

quants (its own "Q4_K_M" of this model actually ships Q5_0/Q8_0). All tiers

therefore use the Q8_0 / Q5_0 / Q4_0 ladder, which is valid on every tensor.

6-bit tiers are not possible for this architecture (the legacy 6-bit type

is not selectable by llama-quantize, and Q6_K requires dimensions this model

does not have), so the family is eight honest tiers — no tier name overstates

its content. The smallest files floor around 19-20 GB. The two exception roles

(attention output and SSM output, ne0=4096) take K-quants at the same size, so

the imatrix still has something to guide.

All tiers share one imatrix (see below) and the embedded MTP head at Q8_0.

Files

| tier | family | ~size |

|---|---|---|

| UD-IQ4_XS | Unsloth Dynamic 2.0 | 20.3 GB |

| UD-Q4_K_XL | Unsloth Dynamic 2.0 | 21.1 GB |

| UD-Q5_K_S | Unsloth Dynamic 2.0 | 24.0 GB |

| APEX-I-Balanced | APEX | 23.4 GB |

| APEX-I-Quality | APEX | 20.8 GB |

| APEX-I-Compact | APEX | 20.3 GB |

| APEX-I-Mini | APEX | 19.3 GB |

| APEX-I-Compact-v2D-lite | APEX | 20.4 GB |

| <model>-imatrix.gguf | importance matrix (980 chunks) | ~0.19 GB |

| sha256sums.txt | checksums of every file, as built | — |

Exact sizes and sha256 are in sha256sums.txt; every file was byte-count

verified against the hub after upload.

imatrix

Calibrated on the model's own output (agentic-coding rollouts) blended 1:1

with bartowski's calibration_datav5, evenly interleaved — 980 chunks on the

bf16 reference. As with every MTP build: the draft head (blk.52) is never

executed during calibration, so it is quantized unguided (here at Q8_0).

Head fingerprint

blk.52.nextn.hnorm.weight  sha256  <filled after build>

Serving

llama-server -m <model>.gguf -c 65536 -fa on --jinja \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 1 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
  --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 24 --spec-ngram-mod-n-match 48

draft-mtp uses the embedded head; ngram-mod covers repeat-heavy output.

(Requires a llama.cpp build with Nemotron MTP support, e.g. ≥ 2026-08-10.)

Building your own tier

hf download ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF \
  NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf --local-dir .
llama-quantize --imatrix <model>-imatrix.gguf \
  --tensor-type-file <tier-map>.txt master.gguf out.gguf <BASE>

Credits

NVIDIA — Nemotron-3.5-Lightning-30B-A3B ·

llama.cpp team — official bf16 master ·

Unsloth — Dynamic 2.0 quant map logic ·

mudler — APEX method ·

bartowski — calibration corpus and

the arch reference imatrix · llama.cpp

License: openmdw-1.1, inherited from the base model.