---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

language:

- en

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

pipeline_tag: text-generation

tags:

- llama.cpp

- gguf

- mtp

- speculative-decoding

- nemotron

- moe

- mamba

- draft-model

---

MTP draft head for NVIDIA Nemotron 3.5 Lightning (GGUF)

Multi-Token Prediction (MTP) draft GGUF for

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16,

for use as a speculative decoding draft with llama.cpp's --spec-type draft-mtp.

The original NVIDIA release contains MTP (NextN) layers. This file restores the MTP head

so llama.cpp users can get speculative decoding on this model.

Contents

• mtp-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.gguf (4.1 GB, F16) — MTP draft model:

the NextN block (dense attention sub-layer + 128-expert MoE sub-layer), token

embeddings, final norm and LM head, exported with convert_hf_to_gguf.py --mtp.

Standalone metadata: nemotron_h_moe, 53 blocks (52 + 1 NextN), 1M context.

Usage (llama.cpp, requires nemotron_h_moe support)

llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf \
  -hfrd mtp-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3

The main model can be any GGUF quant of this model (Q4_K_M, Q6_K, Q8_0, ...) — the

draft head is applied on top of it. Tokenizer/vocab of the draft matches the base model.

Conversion

git clone https://github.com/ggml-org/llama.cpp   # build with nemotron_h_moe support
python3 convert_hf_to_gguf.py <hf-dir> --mtp --outtype f16

Requires only the MTP-bearing safetensors shard

(model-00014-of-00014.safetensors) plus the shards holding token embeddings, final

norm and LM head; a filtered model.safetensors.index.json allows converting without

the full 65 GB BF16 checkpoint.

License

Derived from NVIDIA weights — OpenMDW-1.1, see

LICENSE (attribution: NVIDIA CORPORATION & AFFILIATES).