---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16

tags:

- gguf

- nemotron

- moe

- mamba

- apex

---

Nemotron-Labs-3-Puzzle-75B-A9B — APEX GGUF quants

APEX (per-tensor, imatrix-guided precision allocation) quants of NVIDIA's

Nemotron-Labs-3-Puzzle-75B-A9B,

a hybrid Mamba2 / Attention / MoE model produced by NVIDIA's Puzzle neural

architecture search, distilled down from

Nemotron-3-Super-120B-A12B

(120.7B/12.8B-active) to 75.3B total / 9.3B active parameters.

Requires a patched llama.cpp

nemotron_h_puzzle does not exist in mainline llama.cpp. Puzzle's MoE layers

are genuinely heterogeneous per-layer — different expert widths (1280–2688)

and different top-k values (4–18) from one MoE layer to the next — which

mainline's nemotron_h_moe architecture doesn't support (it assumes one

uniform expert width and top-k for the whole model).

A patch adding real per-layer support (arrays instead of scalars for expert

width and top-k, threaded through both tensor creation and the MoE

aggregation step) is included:

brywil/llama.cpp, puzzle-nemotron branch

(based on mainline llama.cpp, MIT license). The patch was verified

bit-exact against the source model (every tensor, across every block type —

mamba, MoE, attention — diffed to the original checkpoint with zero

deviation, modulo the standard RoPE Q/K permutation every llama.cpp

converter applies) before any quantization was done.

Convert + quantize with the patched build (mainline convert_hf_to_gguf.py +

llama-quantize both work once patched — no special flags beyond the usual

--imatrix/--tensor-type-file).

Quants

| tier | size | bits/weight | coherent | tool calls |

|---|---|---|---|---|

| i-mini | 31.0 GB | ~2.94 | yes | 3/3 |

| i-compact | 39.3 GB | ~3.96 | yes | 3/3 |

| i-quality | 49.9 GB | ~5.30 | yes | 3/3 |

"Coherent" = real-generation check on a fixed prompt suite (not a canned

demo). "Tool calls" = a 3-turn dependent tool-calling chain (with a

distractor tool correctly never invoked), 3/3 trials succeeding at every

tier.

Perplexity (wikitext, wiki.test.raw, 588 chunks, n_ctx=512 — identical

methodology across every row below):

| model | tier | size | PPL |

|---|---|---|---|

| Puzzle-75B-A9B (this repo) | i-mini | 31.0 GB | 6.343 |

| Puzzle-75B-A9B (this repo) | i-compact | 39.3 GB | 5.902 |

| Puzzle-75B-A9B (this repo) | i-quality | 49.9 GB | 5.752 |

| Nemotron-3-Super-120B-A12B (base model, APEX) | i-compact | 67.8 GB | 4.701 |

| Nemotron-3-Super-120B-A12B (base model, APEX) | i-quality | 85.8 GB | 4.639 |

At matched tier names, Puzzle is measurably higher perplexity than the base

model it was distilled from (worse quality) while being roughly 42% smaller

— a real, quantified quality-for-size tradeoff from NVIDIA's Puzzle NAS

compression, not a free lunch. Coherence and tool-calling both still hold

at every tier (see above).

Separately, this model was run through a 5-task hard-tier coding benchmark

(exact-context-window agentic tasks via OpenCode: a buddy allocator, a

batch scheduler, a circuit breaker, an advanced expression parser, and an

existing-codebase bug-fix/extend task) at the i-quality tier: **5/5 tasks

fully passed, 59/59 individual test cases, 100% mean score** — matching a

same-size-class dense model (Qwen3.6-35B-A3B) exactly on the same suite,

albeit taking substantially longer in wall-clock time (see the speed note

below).

One structural note that affects all three tiers equally: 7 of 763 tensors

(the ffn_down_exps weight on the 7 layers whose expert width is 2688 —

not divisible by 256) fall back from the target quant type to a compatible

alternative, since that width doesn't satisfy the block-size requirement of

some low-bit formats. This is a real, minor, disclosed size cost (roughly

1-3GB depending on tier), not a defect.

Known limitation: generation speed

Measured decode speed on this quant is roughly 6.5–9 tokens/second,

notably slower than the base Nemotron-3-Super model (~15-17 tok/s on

mainline, uniform-MoE nemotron_h_moe) despite Puzzle having fewer total

and active parameters. Root cause: ggml's CUDA-graph capture — which

normally cuts per-token kernel-launch overhead substantially during

decode — is disabled by upstream ggml whenever a MoE layer's per-token

expert count exceeds a per-quant-type threshold

(ggml_cuda_graph_check_compability in ggml-cuda.cu; see the TODO in

that function and linked upstream PR discussion). Puzzle's per-layer top-k

reaches 18 on some layers, well past these thresholds for several of the

quant types used here; Super's uniform, smaller top-k stays under them

throughout. This is an actively-tracked upstream ggml limitation, not

something introduced by this patch, and may improve as that work lands.

Calibration

Nemotron-Labs-3-Puzzle-75B-A9B.imatrix (included in this repo) — 126-chunk

calibration run over a diverse corpus (chat, code, reasoning, tool-calling),

392 entries, covering every block type (mamba/attention/MoE) at every

distinct per-layer shape.

Not supported

MTP (multi-token prediction) draft head is dropped during conversion, same

as every other Nemotron-H variant in mainline llama.cpp — no inference

support for it exists upstream.