MRockatansky/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-MTP-GGUF overview

---

license: other

license_name: openmdw-1.1

license_link: https://developer.nvidia.com/open-model-license

base_model:

• nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16

quantized_by: MRockatansky

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- llama.cpp

- nemotron

- nemotron-3

- latent-moe

- speculative-decoding

- mtp

- mamba2

- hybrid

- moe

---

GGUFs made with a non-mainline build of llama.cpp:

nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16 (official), converted with convert_hf_to_gguf.py at

Premshay/llama.cpp@731368b5d (branch nemotron-super-mtp, stacked on PR #25444), quantized with llama-quantize from the same build

The GGUFs should be loaded with the following fork of llama.cpp on the nemotron-super-mtp branch:

https://github.com/Premshay/llama.cpp/tree/nemotron-super-mtp

Just an experiment to try the forked llama.cpp built on top of PR#25444 to support

this model's built in MTP heads. Use the fork of llama.cpp above with --spec-type draft-mtp

enabled in llama-server to provide a nice speedup in token generation.

The contributor of the fork explained their methodology on the model card below. I just did the same but with

Nemotron Puzzle BF16 weights instead of using Nemotron Super:

https://huggingface.co/Premshay/Nemotron-3-Super-120B-A12B-MTP-GGUF