meshllm/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-MTPv2-GGUF overview

---

license: other

library_name: mesh-llm

base_model:

• nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

pipeline_tag: text-generation

tags:

• gguf
• mesh-llm
• skippy
• mtp
• mtpv2
• speculative-decoding
• nemotron
• distributed-inference
• local-inference
• openai-compatible

---

<div align="center">

<a href="https://www.meshllm.cloud">

<img src="https://meshllm.cloud/assets/images/jelly-logo-wordmark.png" alt="Mesh LLM" width="220">

</a>

<h1>NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-MTPv2-GGUF</h1>

<p>

<strong>Composed MTPv2 GGUF artifact for Mesh LLM</strong>

</p>

<p>

<a href="https://www.meshllm.cloud"><img alt="Website" src="https://img.shields.io/badge/Website-meshllm.cloud-111111?style=for-the-badge"></a>

<a href="https://github.com/Mesh-LLM/mesh-llm"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-Mesh--LLM-24292f?style=for-the-badge&logo=github"></a>

<a href="https://discord.gg/rs6fmc63eN"><img alt="Discord" src="https://img.shields.io/badge/Discord-Join-5865F2?style=for-the-badge&logo=discord&logoColor=white"></a>

</p>

</div>

Composite GGUF artifact combining the unsloth UD-Q4_K_XL quantization of

NVIDIA Nemotron-3-Super-120B-A12B with NVIDIA's official **MTPv2 multi-token

prediction head**, spliced in as a native NextN draft block for speculative

decoding with Mesh LLM.

Model Overview

| Property | Value |

|---|---|

| Source target | unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF (UD-Q4_K_XL, 3 shards) |

| Source draft | nvidia/Nemotron-3-Super-120B-A12B-BF16-MTPv2 (BF16 MTP head) |

| Base model | nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 |

| Family | Nemotron 3 (hybrid SSM + latent MoE) |

| GGUF architecture | nemotron_h_moe |

| Quantization | UD-Q4_K_XL (target) + BF16 (draft block) |

| Layers | 88 target + 1 MTP NextN block (blk.88) |

| MTP metadata | nemotron_h_moe.nextn_predict_layers=1 |

| Context length | 1,048,576 |

| Artifact size | ~84 GiB across 3 shards |

What Is Included

| Artifact | Size | Contents |

|---|---|---|

| ...-00001-of-00003.gguf | 7 MB | Metadata shard (rewritten: block_count 88→89, nextn_predict_layers=1) |

| ...-00002-of-00003.gguf | 46.5 GiB | Middle shard, byte-identical to the source |

| ...-00003-of-00003.gguf | 37.0 GiB | Last shard recomposed with the BF16 MTP draft tensors appended |

How It Was Built

The target's quantized shards are kept untouched wherever possible; only shard 1

(metadata) and shard 3 (draft append) are rewritten. NVIDIA's BF16 MTPv2 head is

converted with the pinned llama.cpp converter (convert_hf_to_gguf.py --mtp),

which folds the two MTP layers into a single blk.88 NextN block, then spliced

past the target tensors with skippy-quantize compose-mtp (byte-copied

verbatim).

• Pipeline: skippy-quantize compose-mtp + validate-mtp-attach
• mesh-llm revision: bbd84c238311562f968bbbcaf3a8b2727a6ac642 (PR #1439)
• Build job: meshllm/6a8ed10345686a1580c0a52b

Validation

validate-mtp-attach passed on the published composite in the build job: the

3-part model loads and the MTP draft attaches natively (parts=3, layers=89,

mtp_layers=1, session created).

Recommended Use

• Local and private inference with Mesh LLM, using Skippy split serving across

machines (the ~84 GiB artifact is designed for sharded serving — see

docs/SKIPPY_SPLITS.md).

• Native Nemotron MTPv2 speculative decoding: one ordinary next token plus one

MTP-predicted token per decode step, verified against the target.

• OpenAI-compatible chat/completions workflows through Mesh LLM's local API.

Quickstart

# Download the composite (all 3 shards).
hf download meshllm/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-MTPv2-GGUF \
  --include '*-MTPv2-0000*-of-00003.gguf'

# Run locally with Mesh LLM (split serving across your machines).
mesh-llm serve --model "meshllm/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-MTPv2-GGUF"

# Check the local OpenAI-compatible model list.
curl -s http://localhost:3131/v1/models

Why MTP Matters

MTP (multi-token prediction) lets the model produce a next token and an extra

predicted token in the same decode step. The runtime uses the extra token as a

speculative candidate and verifies it against the target instead of waiting for

a full model pass per token — higher throughput at the same quality, since every

accepted candidate is verified by the target itself.

License

This artifact derives from NVIDIA's Nemotron-3-Super-120B-A12B weights and

unsloth's UD-Q4_K_XL quantization of them. License terms follow the source

repositories — see

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

and

unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF.

For architecture details, chat template guidance, and sampling recommendations,

see those model cards.