---

license: openmdw-1.1

language:

• en
• es
• fr
• de
• it
• ja

tags:

• gguf
• llama.cpp
• rocmfp4
• amd
• vulkan
• moe
• mamba2
• hybrid

pipeline_tag: text-generation

base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16

---

NVIDIA Nemotron 3.5 Lightning 30B-A3B — ROCmFP4 GGUF

Quantized GGUF conversions of NVIDIA Nemotron 3.5 Lightning 30B-A3B Base (BF16)

using the experimental ROCmFPX quantization family from the

ROCmFPX llama.cpp fork.

> IMPORTANT: These files use custom Q4_0_ROCMFP4_* GGUF tensor types.

> They are not loadable by mainline llama.cpp. You must build/run the

> ROCmFPX fork (see how_to_use.md). They are tuned and

> validated on AMD Strix Halo / RDNA3.5 (gfx1151), with HIP/ROCm and Vulkan

> kernel support (plus CPU reference paths).

>

> Note on Hugging Face Hub metadata: The Hub warning *"We're not able to

> determine the quantization variants" is expected because Q4_0_ROCMFP4_

> are experimental, non-standard GGUF types. The table below is authoritative.

Files

| File | Preset | bpw | Size | Notes |

|------|--------|----:|-----:|-------|

| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN.gguf | Q4_0_ROCMFP4_STRIX_LEAN | ~4.38 | 15.73 GiB | Recommended. Strix Halo K/V recipe + Q5_K token embeddings |

| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-FAST.gguf | Q4_0_ROCMFP4_FAST | ~4.25 | 15.66 GiB | Speed-first single-scale layout |

| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-COHERENT.gguf | Q4_0_ROCMFP4_COHERENT | ~4.70 | 16.74 GiB | Agent/tool/JSON/code preset (protected embeddings + Q8_0) |

All files are quantized from the BF16 source (recommended quality path) and

advertise context_length = 262144 (the model's real max_position_embeddings).

*Note: Token embeddings for 2688 hidden dimensions fall back from Q5_K/Q6_K

to Q5_1/Q8_0 because 2688 is not divisible by 256.*

Model overview

• Architecture: nemotron_h_moe (hybrid Mamba2 / Attention / MoE)
• 52 layers: 23 Mamba2 + 6 Attention + 23 MoE
• 128 routed experts, 6 active (A3B), 1 shared expert
• ~31.6B total params (3.5B active per token, A3B)
• Vocabulary: 131072 · Context: 262144 (256K)
• License: OpenMDW-1.1 (see LICENSE)

Quality & Benchmarks

| Preset | Vulkan0 Prompt (pp512) | Vulkan0 Decode (tg128) | ROCm0 Prompt (pp512) | ROCm0 Decode (tg128) |

|:-------|:----------------------:|:---------------------:|:-------------------:|:-------------------:|

| FAST (Q4_0_ROCMFP4_FAST) | 1310.5 t/s | 86.0 t/s | 1079.3 t/s | 80.3 t/s |

| STRIX_LEAN (Q4_0_ROCMFP4_STRIX_LEAN) | 1299.7 t/s | 85.6 t/s | 1075.4 t/s | 79.4 t/s |

| COHERENT (Q4_0_ROCMFP4_COHERENT) | 1290.4 t/s | 81.6 t/s | 1302.2 t/s | 77.8 t/s |

Perplexity: STRIX_LEAN scores 5.9936 ± 0.0358 on wikitext-2.

Measured on Framework AMD Strix Halo (128 GB unified RAM, gfx1151, ROCm 7.2.3).

Quick start

See how_to_use.md for full instructions (build + run).

# 1. Build the ROCmFPX fork
git clone https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX && env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh   # Strix Halo

# 2. Run on Vulkan (recommended on Strix Halo)
build-strix-rocmfp4/bin/llama-completion -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN.gguf \
  -p "What is 2+2?" -n 64 -dev Vulkan0 -ngl 999 -fa on -c 8192

# 3. Or on HIP/ROCm (unified memory enables APUs/iGPUs)
HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
  build-strix-rocmfp4/bin/llama-completion -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN.gguf \
  -p "What is 2+2?" -n 64 -dev ROCm0 -ngl 999 -fa on -c 8192

Technical Notes & Findings

• MTP/NextN head is not included in this conversion (the NemotronH converter

skips mtp.* tensors for MoE models).

• NVFP4 path analysis: We evaluated converting the native NVFP4 checkpoint.

With our converter patches, native NVFP4 GGUF loads, but scores PPL 109.79

because runtime kernels do not integrate ModelOpt's companion scale2 factor

(~1.4e-4). The clean BF16 → ROCmFP4 path is used for all delivered models.

• Converter patches, reproduction scripts, and full benchmark notes are available in

the companion repo: julianmb/nemotron-3.5-30b-a3b-rocmfp4 on GitHub.

Credits

• Model: NVIDIA (weights, architecture, license: OpenMDW-1.1)
• ROCmFPX / ROCmFP4 quantization + kernels: charlie12345/ROCmFPX
• Conversion & quantization performed by: julianmb