---

license: apache-2.0

base_model: nex-agi/Nex-N2-mini

quantized_by: LibertAIDAI

tags:

- gguf

- llama.cpp

- nvfp4

- blackwell

- nex-n2

- moe

- multimodal

language:

- en

pipeline_tag: image-text-to-text

---

Nex-N2-mini NVFP4 GGUF

NVFP4 GGUF quantizations of nex-agi/Nex-N2-mini, produced for use with llama.cpp.

This is a MoE model — 35B total parameters, ~3B activated per token (8 of 256 experts). The expert FFN tensors — both routed experts (_exps) and shared experts (_shexp), 240 tensors total — are quantized to NVFP4 (NVIDIA's 4-bit float with E4M3 block scales), repacked from the calibrated r0b0tlab/nex-n2-mini-nvfp4 checkpoint (NVIDIA ModelOpt v0.44). Because the experts dominate the model's memory footprint, NVFP4-quantizing them gives most of the size reduction; the remaining tensors (attention, linear-attention blocks, embeddings) use a conventional GGUF quant.

About LibertAI

LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.

If you want to put this model (or any other) to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.

> Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200), llama.cpp uses native NVFP4 tensor-core MMA kernels (added in llama.cpp #22196) for the expert matmul — the dominant compute cost during MoE inference. On older GPUs the path falls back to dp4a/MMQ kernels, where these GGUFs run but offer no perf advantage over standard K-quants.

Files

| File | Size | Experts | Other tensors | When to pick |

|------|------|---------|---------------|--------------|

| Nex-N2-mini-NVFP4-Q4_K_M.gguf | 19.8 GB | NVFP4 | Q4_K_M (imatrix) | Recommended — smallest and fastest for serving on Blackwell |

| Nex-N2-mini-NVFP4-Q8_0.gguf | 20.7 GB | NVFP4 | Q8_0 | Higher quality non-expert tensors |

| Nex-N2-mini-NVFP4-BF16.gguf | 22.9 GB | NVFP4 | BF16 | Max quality (preserves source precision for non-expert tensors) |

| mmproj-Nex-N2-mini-F16.gguf | 903 MB | — | F16 vision tower | Required for image input — reusable with any Nex-N2-mini GGUF |

Performance

Measured on an NVIDIA RTX 5090 (32 GB, Blackwell, sm_120), llama.cpp build 85f99dca8.

Variant comparison (single-stream, llama-bench 512 in / 64 out)

| Variant | Size | PP512 (tok/s) | TG64 (tok/s) |

|---------|------|---------------|--------------|

| NVFP4-Q4_K_M | 18.41 GiB | 9514 | 259 |

| NVFP4-Q8_0 | 19.30 GiB | 10096 | 234 |

| NVFP4-BF16 | 21.31 GiB | 9678 | 193 |

Batched serving vs stock Q4_K_M (honest comparison)

llama-batched-bench, 512 in / 128 out per stream, vs bartowski/nex-agi_Nex-N2-mini-GGUF Q4_K_M (19.91 GiB):

| Parallel streams | Stock Q4_K_M (total tok/s) | NVFP4-Q4_K_M (total tok/s) | Delta |

|---|---|---|---|

| 1 | 1255 | 1190 | −5% |

| 4 | 2473 | 2549 | +3% |

| 8 | 3159 | 3314 | +5% |

| 16 | 3274 | 3447 | +5% |

Prompt processing is ~10% faster on NVFP4 at every batch size (~9900–10200 vs ~8600–9300 tok/s). Single-stream decode still slightly favors stock Q4_K_M's MMQ kernel; from 4 concurrent streams up — the realistic serving regime — the NVFP4 variant wins on total throughput while using 1.5 GiB less VRAM.

(This is the first MoE release where our NVFP4 expert path beats stock K-quants in batched serving — earlier this year the MMQ kernel still won; upstream NVFP4 MoE optimization has since closed the gap.)

Usage

Text-only (CLI)

llama-cli -m Nex-N2-mini-NVFP4-Q4_K_M.gguf -ngl 999 -c 8192 -p "Your prompt here"

Multimodal (server, vision + text)

llama-server \
  -m Nex-N2-mini-NVFP4-Q4_K_M.gguf \
  --mmproj mmproj-Nex-N2-mini-F16.gguf \
  -ngl 999 -c 32768 \
  --host 0.0.0.0 --port 8080

Then POST to /v1/chat/completions with image content blocks — see the llama.cpp multimodal docs.

Thinking model — fixed chat template included

Nex-N2 uses "Agentic Thinking" with adaptive reasoning depth — the chat template enables <think> blocks by default.

These GGUFs embed a fixed chat template. The upstream nex-agi template prefills the assistant turn with '<think>' (no trailing newline) while rendering past assistant reasoning as '<think>\n…'. That inconsistency breaks llama.cpp's reasoning extraction: the parser never recognizes the forced-open think block, so the full chain-of-thought (and a stray </think>) leaks into content instead of reasoning_content — on every llama.cpp build, regardless of --reasoning-format. Other community GGUFs of this model embed the upstream template and inherit the bug. Our embedded template adds the missing newline, so reasoning_content / content separation and tool-call parsing work out of the box with stock llama-server --jinja.

About the architecture

Nex-N2-mini is built on the Qwen3.5-MoE architecture (qwen35moe in GGUF): a hybrid linear-attention MoE with 40 layers (3 of every 4 layers use linear attention, every 4th is full attention), 256 experts (8 active per token) plus a shared expert, totalling 35B parameters with ~3B active. The upstream config declares a 1-layer MTP head, but the published checkpoints do not include MTP weights, so no MTP/speculative variant can be produced from public weights. The ModelOpt source keeps attention projections, linear-attention blocks, embeddings, and lm_head at BF16 — routed + shared expert FFNs (40 layers × 6 tensors) are NVFP4. The variants above differ only in how those non-expert tensors are stored.

Sources & credits

• Base model: nex-agi/Nex-N2-mini by Nex AGI — Apache 2.0
• NVFP4 calibration source: r0b0tlab/nex-n2-mini-nvfp4 (NVIDIA ModelOpt v0.44)
• mmproj source: official BF16 weights from nex-agi/Nex-N2-mini
• Stock baseline for benchmarks: bartowski/nex-agi_Nex-N2-mini-GGUF
• Regular (non-NVFP4) imatrix quants with the same template fix: LibertAIDAI/Nex-N2-mini-GGUF
• Tooling: llama.cpp convert_hf_to_gguf.py and llama-quantize

License

Apache 2.0, inherited from the upstream model.