---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model:

• nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
• nvidia/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning

pipeline_tag: image-text-to-text

tags:

• gguf
• llama.cpp
• multimodal
• audio
• video
• nemotron
• agents

---

Nemotron-3.5-Lightning-Omni-30B-A3B-GGUF

NVIDIA released Nemotron 3.5 Lightning as a text-only agent model. This

repository is Lightning with eyes and ears: the C-RADIO vision tower and the

Parakeet audio tower from NVIDIA's own Nemotron-3-Nano-Omni, attached through

their unified projector, quantized for llama.cpp, and tested modality by

modality before upload. Image, audio, and video understanding are the

released, gated capabilities. One-pass video WITH its soundtrack also works

zero-shot; there is one prompting note (below) for getting both channels

reliably in a single answer.

No training was involved, and that is the interesting part: Lightning shares

the exact backbone geometry of the model those projectors were trained for,

so perception transfers zero-shot. Details in How This Was Made below.

Model Summary

| | |

|---|---|

| Base LLM | NVIDIA Nemotron 3.5 Lightning 30B-A3B (hybrid Mamba-2 + MoE + attention, 3B active, 1M context) |

| Vision tower | C-RADIO ViT-H, lifted from Nemotron-3-Nano-Omni |

| Audio tower | Parakeet FastConformer-XL encoder, lifted from Nemotron-3-Nano-Omni |

| Video | C-RADIO per frame + temporal 2-frame patches + EVS token pruning |

| Projector | one unified mmproj file, both towers, upstream Parakeet tensor layout |

| Modalities in | text, image, audio, video, and one-pass video+audio (all released; one-pass has a prompting note) |

| Runs on | llama.cpp (see compatibility matrix below), from Jetson to DGX Spark to datacenter |

Quick Start

Download a quant plus the projector, then:

./llama-server -m Nemotron-3.5-Lightning-Omni-30B-A3B-Q4_K_M.gguf --mmproj mmproj-omni-vision-audio-v2-F16.gguf

Open http://localhost:8080 and drag in a photo, a wav, or an mp4. The same

endpoint serves /v1/chat/completions with image_url, input_audio, and video

content parts.

llama.cpp compatibility

| Capability | Stock llama.cpp (current master) | llama.cpp-omni fork b10598 |

|---|---|---|

| Text | yes | yes |

| Image | yes | yes |

| Audio | yes | yes |

| Video | no for Nemotron models (upstream's video path covers Qwen-VL style; the Nemotron temporal+EVS graph is fork-only) | yes, gated per quant |

| Video with soundtrack, one pass | no for Nemotron models | yes; ask for both channels explicitly, see Verification |

Prebuilt fork binaries for Linux arm64, Linux x64, Win x64:

https://github.com/engram-ae/llama.cpp-omni/releases/tag/b10598

Files

Quantizations land one by one as each passes its gates; this table and

TEST-MATRIX.md grow together until all nine tiers are present.

<!-- FILES-TABLE-START -->

| File | Size | Verified | Video |

|---|---|---|---|

| Q3_K_M | 20.7 GB | gated: text, image, audio | see matrix |

| IQ3_M | 18.9 GB | gated: text, image, audio | see matrix |

| IQ4_XS | 19.0 GB | gated: text, image, audio | see matrix |

| MXFP4_MOE | 18.7 GB | gated: text, image, audio | see matrix |

| Q4_K_M | 25.4 GB | gated: text, image, audio | see matrix |

| Q5_K_M | 27.0 GB | gated: text, image, audio | see matrix |

| Q6_K | 34.9 GB | gated: text, image, audio | see matrix |

| Q8_0 | 35.0 GB | gated: text, image, audio | see matrix |

| mmproj (unified, all tiers) | 3.0 GB | both towers | required |

<!-- FILES-TABLE-END -->

The unified projector mmproj-omni-vision-audio-v2-F16.gguf serves every

tier; single-tower use is possible but the unified file is the intended

path.

Every quantization in this table passed the text, image, audio, and

silent-video gates before upload. The one-pass audio+video column is

recorded per quant in TEST-MATRIX.md and is

experimental in this revision.

Verification

Claims are cheap; here is what was measured, on a DGX Spark (GB10):

• Five gates per quantization at temperature 0, graded against known ground

truths, published in TEST-MATRIX.md. Text, image, and

audio must pass for a file to ship; the video columns are recorded

honestly and do not block, because video is experimental here.

• One-pass A/V, the measured truth (2026-08-25, corrected twice as

evidence came in): first hypothesis was temporal-pathway drift, killed

by measurement. Second hypothesis was that audio irreversibly dominates,

also killed by measurement. What actually happens: video-only works at

every length (2 to 64 frames, matches the donor), and in one pass WITH

a soundtrack both channels are fully present and attendable. With a

NEUTRAL single-subject prompt ("what is in the video?") zero-shot

Lightning tends to answer from the audio and under-report the picture.

With a prompt that asks for BOTH, it returns both correctly, every time

tested. Two verified examples, temp 0: "Describe what is shown on screen

AND transcribe what the narrator says" and "Give me two things: (1) the

on-screen visuals, (2) the spoken words" each produced the full visual

description AND the full transcription in a single answer. So one-pass

A/V is a real, released capability with a prompting note, not a broken

feature. The donor Omni, co-trained on interleaved A/V, balances both

even at a neutral prompt; a future Lightning projector tune could shift

the neutral-prompt default the same way, but it is polish, not a fix for

something broken.

• ASR transfer cost, measured: on the 38-word gate narration, the donor

Omni model scores 2.6 percent word error rate (one proper noun); this

zero-shot assembly scores 7.9 percent (three proper nouns, including the

same one the donor misses). Single-clip gate, stated at exactly that

scope, not a benchmark.

• Decode speed, measured on the same machine and binary:

| Model (same binary, DGX Spark GB10) | Decode |

|---|---|

| Lightning-Omni, community UD-Q4_K_M flavor | 61.1 tok/s |

| Donor Nemotron-3-Omni, Q4_K_M | 68.7 tok/s |

The projector adds no decode cost (it runs once at encode). Numbers for

this repository's own quantizations land here as they are measured.

• The underlying video and audio inference graphs were validated against

NVIDIA's PyTorch reference during the fork's development (video 0.0019

percent relative L2 with exact EVS parity; audio transcribing the

reference narration word for word). See the fork repository for that

work: https://github.com/engram-ae/llama.cpp-omni

Benchmarks

The language model weights are byte-identical to NVIDIA's release, so

NVIDIA's published text benchmarks describe this model's brain at BF16:

SWE-bench Verified 51.56, GPQA Diamond 75.44, MMLU Pro 81.94 (their

numbers, their card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16).

Quantization effects on those scores are not measured here; the BF16 file

in this repository is the reference conversion.

Multimodal quality is what this repository adds, and it is gated rather

than benchmarked: see TEST-MATRIX.md and the measured ASR note above.

Model Architecture

The assembly, exactly:

| Part | Geometry | Origin |

|---|---|---|

| Language backbone | nemotron_h hybrid Mamba-2/MoE/attention, hidden 2688, 52 layers, vocab 131072, 30B total / 3B active | NVIDIA Nemotron 3.5 Lightning, unmodified |

| Vision tower | C-RADIO ViT-H, patch 16, 512px, hidden 1280 | Nemotron-3-Nano-Omni |

| Audio tower | Parakeet FastConformer-XL, 24 layers, hidden 1024, 128 mel | Nemotron-3-Nano-Omni |

| Video temporal | 2-frame channel-concat patch embedding, stored as v.patch_embd.weight.1 (16x16x6x1280), plus EVS token pruning | Nemotron-3-Nano-Omni |

| Projectors | vision: pixel-shuffle then MLP to 2688; audio: MLP to 2688; both in one mmproj | Nemotron-3-Nano-Omni |

How This Was Made (and why no training was needed)

NVIDIA trained the Nano-Omni projectors to translate C-RADIO and Parakeet

features into the embedding space of a nemotron_h backbone with hidden size

2688, 52 layers, and a 131072 vocabulary. Nemotron 3.5 Lightning has exactly

that geometry: it is a sibling checkpoint of the same architecture family.

Attaching the projector to Lightning is therefore a compatible operation,

and it works zero-shot: the perception pathway transfers, and the measured

cost is two additional word errors on the 38-word ASR gate (see

Verification), concentrated in rare proper nouns. Image understanding

matches the donor exactly; video is where the drift concentrates, which is

why it ships as experimental here.

A projector fine-tune against Lightning specifically is the known path to

closing that gap and is planned as a follow-up revision.

Input

Text prompts; images (png/jpg/webp); audio (wav/mp3); video files (mp4 and

other ffmpeg-readable containers), with or without a soundtrack. One media

file per turn.

Output

Text.

Software Integration

• Runtime: llama.cpp. Full modality support in the llama.cpp-omni fork,

release b10598 (prebuilt Linux arm64 CUDA 13 binaries included). Text,

image, and audio also run on stock llama.cpp current master with these

same files.

• Tested on: NVIDIA DGX Spark (GB10, CUDA 13, sm_121).

License / Terms of Use

• Language model weights: NVIDIA Nemotron 3.5 Lightning, released by NVIDIA

under the Open Model, Data and Weights License 1.1 (OpenMDW-1.1):

https://openmdw.ai/license/1-1/

• Encoder towers and projector weights: lifted from NVIDIA

Nemotron-3-Nano-Omni, governed by the NVIDIA Open Model License terms of

that release.

• The GGUF conversion, the quantizations, the unified projector packaging,

and the llama.cpp video/one-pass graphs are my work; the underlying

weights remain NVIDIA's under their respective licenses.

Model Version(s)

• v1 (2026-08): zero-shot assembly, nine quantizations, five-gate matrix.
• Planned v1.1: Lightning-specific projector fine-tune (closes the measured

ASR proper-noun gap and the experimental-video seam).

Known Limitations

• One-pass audio+video works, with a prompting note: a neutral

single-subject prompt may answer from the audio and under-report the

picture. Ask for both channels explicitly (see Verification) and it

returns both correctly. Video-only and audio-only are unconditionally

reliable.

• ASR smudges rare proper nouns more than the donor Omni does (measured

above). General transcription is faithful.

• Multimodal gates are functional checks, not benchmark suites. Treat

TEST-MATRIX.md as exactly what it is: proof each file works, per

modality, on real inputs.

• Video and one-pass A/V require the fork binaries until the planned

upstream PR lands.

Release Date

2026-08 (Hugging Face).