---

license: other

license_name: nvidia-oneway-noncommercial

license_link: https://huggingface.co/nvidia/Nemotron-Labs-Audex-30B-A3B/blob/main/license/NVIDIA-OneWay-Noncommercial-License.docx

base_model: nvidia/Nemotron-Labs-Audex-30B-A3B

base_model_relation: quantized

library_name: llama.cpp

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- audex

- nemotron

- audio

- text-generation

- non-commercial

---

Nemotron-Labs-Audex-30B-A3B — GGUF

GGUF quantizations of NVIDIA Nemotron-Labs-Audex-30B-A3B (source revision 48616f3d9d5ad960a4d03dc66f2178daf77e7b5f) — a 30B-parameter hybrid Nemotron-H MoE model (~3B active parameters per token, nemotron_h_moe architecture, 1,048,576-token context) with audio understanding and generation capabilities.

Every quant was measured for quality (BF16-referenced KL divergence / perplexity over three corpora) and throughput (llama-bench on 2× RTX PRO 6000 Blackwell Max-Q). Raw data lives in metrics/.

> ⚠️ License: NVIDIA OneWay Noncommercial. These files are derived from NVIDIA Nemotron-Labs-Audex-30B-A3B and are not for commercial use. Audex also includes or derives from third-party components with additional notices. See LICENSE, license/NVIDIA-OneWay-Noncommercial-License.docx, and license/THIRD_PARTY_NOTICES.md.

Repository Layout — Two Release Tracks

| Track | Directory | What it is | Use with |

|---|---|---|---|

| Text-only | quants/ | GGUFs from the materialized Audex text-only checkpoint | Plain llama.cpp text generation |

| Audiogen | audio_quants/ | Full-vocab GGUFs for audio-token generation | The audio_support/ sidecar bundle |

The audio encoder/vocoder assets are not embedded as GGUF tensors — audio workflows need the audio_support/ sidecar (NV-Whisper + audio preprocessor path, causal speech decoder, enhancement VAE, and NVIDIA's HF/vLLM inference scripts).

Which File Should I Download?

For plain text work, use quants/. For audio-token generation, use audio_quants/ + audio_support/.

| Use case | File | Why |

|---|---|---|

| Near-lossless | Q8_0 | Mean KLD 0.006 (text) / 0.008 (audio), ~97% top-token agreement |

| Recommended default | Q5_K_M / Q5_K_L | ~95.5% top-token agreement at ~24 GiB, faster than Q8_0 |

| Smallest solid quant | Q4_K_M / Q4_K_L | ~94.5–94.7% top-token agreement at ~23 GiB, fastest K-quants |

| Experimental | MXFP4_MOE (text-only) | Smallest file (16.7 GiB) and by far the fastest prompt processing, but measurably lower fidelity |

Download

# Text-only quant
hf download LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
  quants/audex-30b-a3b-textonly-Q5_K_M.gguf --local-dir ./audex-gguf

# Audiogen quant + audio sidecar bundle
hf download LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
  audio_quants/audex-30b-a3b-audiogen-Q5_K_M.gguf --local-dir ./audex-gguf
hf download LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
  --include "audio_support/*" --local-dir ./audex-gguf

Verify downloads against SHA256SUMS / AUDIO_SHA256SUMS (per-file .sha256 files are also next to each GGUF).

Run (Text) with llama.cpp

llama.cpp supports the nemotron_h language-model architecture used by these checkpoints. The chat template is embedded in the GGUF.

llama-server \
  --hf-repo LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
  --hf-file quants/audex-30b-a3b-textonly-Q5_K_M.gguf \
  -ngl -1 -sm layer \
  -c 16384

For audio generation, audio QA, and unified speech-to-speech flows, use NVIDIA's inference scripts shipped in the sidecar:

• audio_support/inference_scripts_hf/ — HF transformers path (inference_hf.py, inference_example.sh)
• audio_support/inference_scripts_vllm/audiogen_scripts/ — vLLM TTS / text-to-audio
• audio_support/inference_scripts_vllm/audioqa_scripts/ — vLLM audio QA server + client
• audio_support/inference_scripts_vllm/unified_s2s_scripts/ — cascaded speech-to-speech web demo

Note: llama.cpp does not currently package the full HF NemotronHAudexForConditionalGeneration audio pipeline as a single all-in-one GGUF runtime.

Text-Only Quants (quants/)

Quality = mean over three eval slices (WikiText-2, code, GSM8K); throughput from llama-bench (settings below). BF16 baseline included for reference.

| Quant | Size GiB | BPW | Mean KLD | Same top-p | Prompt tok/s | Gen tok/s |

| --- | ---: | ---: | ---: | ---: | ---: | ---: |

| Q8_0 | 31.3 | 8.51 | 0.00561 | 97.03% | 8,500 | 287.4 |

| Q6_K | 31.2 | 8.49* | 0.00660 | 96.85% | 8,518 | 290.1 |

| Q5_K_L | 24.3 | 6.62 | 0.01243 | 95.55% | 8,245 | 334.2 |

| Q5_K_M | 24.2 | 6.60 | 0.01268 | 95.50% | 8,194 | 333.9 |

| Q4_K_L | 22.9 | 6.24 | 0.01749 | 94.73% | 8,640 | 345.5 |

| Q4_K_M | 22.8 | 6.21 | 0.01799 | 94.73% | 8,579 | 345.2 |

| MXFP4_MOE | 16.7 | 4.55 | 0.04048 | 92.30% | 11,486 | 329.2 |

| (BF16 reference) | ~59 | 16 | 0 | 100% | 4,378 | 176.9 |

• Q6_K_L is also included but is byte-identical to Q6_K (K-quant fallbacks make the L variant resolve to the same tensors).
• MXFP4_MOE is experimental (0 fallback tensors, so it hits its native 4.55 BPW).
• \* Q6_K BPW is inflated by fallback quantization on incompatible tensor shapes — see Caveats.

<details>

<summary>Per-corpus quality breakdown (text-only)</summary>

| Quant | WikiText-2 KLD | WikiText-2 same-top-p | Code KLD | Code same-top-p | GSM8K KLD | GSM8K same-top-p |

| --- | ---: | ---: | ---: | ---: | ---: | ---: |

| Q8_0 | 0.00644 | 96.03% | 0.00609 | 97.27% | 0.00432 | 97.81% |

| Q6_K | 0.00762 | 95.74% | 0.00701 | 97.17% | 0.00517 | 97.64% |

| Q5_K_L | 0.01465 | 94.11% | 0.01244 | 95.87% | 0.01019 | 96.68% |

| Q5_K_M | 0.01524 | 94.03% | 0.01212 | 95.78% | 0.01069 | 96.70% |

| Q4_K_L | 0.02142 | 92.90% | 0.01704 | 95.23% | 0.01402 | 96.06% |

| Q4_K_M | 0.02165 | 92.95% | 0.01774 | 95.15% | 0.01459 | 96.10% |

| MXFP4_MOE | 0.04963 | 89.50% | 0.03931 | 93.17% | 0.03250 | 94.22% |

Base (BF16) perplexities: WikiText-2 10.97, code 2.96, GSM8K 3.38. Full data: metrics/kld_ppl_summary.tsv, metrics/kld_ppl_metrics.json.

</details>

Audiogen Quants (audio_quants/)

Full-vocab audiogen language-model path — use together with audio_support/.

| Quant | Size GiB | BPW | Mean KLD | Same top-p | Prompt tok/s | Gen tok/s |

| --- | ---: | ---: | ---: | ---: | ---: | ---: |

| Q8_0 | 31.7 | 8.51 | 0.00754 | 96.92% | 8,339 | 276.5 |

| Q6_K | 31.6 | 8.49 | 0.00934 | 96.68% | 8,350 | 279.4 |

| Q5_K_L | 24.7 | 6.65 | 0.01599 | 95.33% | 8,044 | 319.4 |

| Q5_K_M | 24.6 | 6.60 | 0.01672 | 95.27% | 8,077 | 319.5 |

| Q4_K_L | 23.3 | 6.27 | 0.02307 | 94.52% | 8,526 | 329.3 |

| Q4_K_M | 23.1 | 6.22 | 0.02342 | 94.54% | 8,447 | 329.6 |

| (BF16 reference) | ~60 | 16 | 0 | 100% | 4,285 | 169.6 |

<details>

<summary>Per-corpus quality breakdown (audiogen)</summary>

| Quant | WikiText-2 KLD | WikiText-2 same-top-p | Code KLD | Code same-top-p | GSM8K KLD | GSM8K same-top-p |

| --- | ---: | ---: | ---: | ---: | ---: | ---: |

| Q8_0 | 0.01039 | 95.83% | 0.00758 | 97.18% | 0.00465 | 97.75% |

| Q6_K | 0.01302 | 95.42% | 0.00928 | 97.03% | 0.00572 | 97.58% |

| Q5_K_L | 0.02190 | 93.66% | 0.01528 | 95.74% | 0.01079 | 96.60% |

| Q5_K_M | 0.02339 | 93.56% | 0.01528 | 95.64% | 0.01150 | 96.62% |

| Q4_K_L | 0.03279 | 92.49% | 0.02128 | 95.09% | 0.01514 | 95.97% |

| Q4_K_M | 0.03266 | 92.59% | 0.02197 | 94.99% | 0.01564 | 96.04% |

Full data: metrics/audio_kld_ppl_summary.tsv, metrics/audio_kld_ppl_metrics.json.

</details>

Charts

!Size vs quality trade-off

!Mean KLD

!Top-token agreement

!Per-corpus KLD

!Generation throughput

!Prompt throughput

Audio Support Bundle (audio_support/)

Common sidecar for the audiogen GGUFs — 58 files, ~7.67 GB. Manifest: audio_support/audio_support_manifest.json; checksums: audio_support/SHA256SUMS.

| Component | Purpose |

|---|---|

| checkpoint_folder_full/ | Audio tower weights (model-audio.safetensors), audio preprocessor, and the full Audex custom configuration/modeling code |

| nv-whisper/ | NV-Whisper encoder used on the audio input path |

| audex_causal_speech_decoder/ | Streaming causal speech decoder |

| enhancement_VAE/ | XCodec RVQ4 enhancement VAE (48 kHz audio enhancement) |

| inference_scripts_hf/, inference_scripts_vllm/ | NVIDIA reference scripts for audio generation, audio QA, and unified speech-to-speech |

Methodology

Quality — BF16-referenced quant-fidelity via llama-perplexity --kl-divergence, 64 chunks × 512-token context per slice, over three corpora: WikiText-2 raw test, a Magicoder-style code slice, and a GSM8K text slice. Imatrix statistics: metrics/imatrix_statistics.txt; imatrix file: gguf/audex-30b-a3b-textonly.imatrix.gguf.

Throughput — llama-bench on 2× NVIDIA RTX PRO 6000 Blackwell Max-Q: prompt 512, generation 256, batch/ubatch 512, 3 repetitions, -ngl -1 -sm layer.

Quant plans — configs/quant_plan.json, configs/audio_quant_plan.json. Release manifests: release_manifest.json, audio_release_manifest.json.

Caveats

• K-quant fallbacks: 133–134 of 401 tensors required fallback quantization because several Nemotron-H MoE/SSM tensor shapes are incompatible with the requested K-quant type. This is why Q6_K lands near 8.5 effective BPW and Q6_K_L is byte-identical to Q6_K. Details are in the artifact summary TSVs and per-quant logs.
• BF16 references not uploaded: the BF16 GGUFs used for KLD/PPL/TPS measurement exceed Hugging Face's 50 GB single-file limit; published artifacts are the quants, metrics, manifests, checksums, licenses, and the audio sidecar.
• No all-in-one audio GGUF: llama.cpp runs the nemotron_h LM; the full audio pipeline requires the sidecar + NVIDIA's HF/vLLM scripts.
• Quant-fidelity ≠ task benchmarks: upstream Audex benchmark scores are reported by NVIDIA on the base model card. The numbers here measure how faithfully each GGUF reproduces the BF16 model, plus llama.cpp throughput.

Acknowledgements

• NVIDIA for Nemotron-Labs-Audex-30B-A3B
• The llama.cpp project