---

license: other

license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE

base_model: neopolita/Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- moe

- pruning

- quantization

- mixture-of-experts

---

!Niwaki

Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-gguf

**GGUF builds of Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx —

Qwen3.8-Flash-Next pruned to 99B total / ~5.4B active parameters — for

llama.cpp on CUDA, Vulkan, Metal and CPU. Requires the

neopolita-llama.cpp fork

(see Usage): stock llama.cpp cannot represent this model's layout yet.**

Niwaki (庭木): routed experts pruned using statistics of their routed tokens

over a mixed web/code/chat/reasoning corpus, reconstructed to compensate, then

briefly distilled from the full model, and stored at low precision.

A paper with the full method is coming soon.

Files

| file | size | wt2 ppl (llama.cpp, 512-ctx) |

|---|---|---|

| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf (recommended) | 42.5 GB | 8.79 ±0.06 |

| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q4K.gguf | 54.1 GB | 8.61 ±0.06 |

Protocol: llama-perplexity, WikiText-2 test, 512-token windows. These

llama.cpp numbers are not directly comparable to the MLX repo's 2048-window

benchmarks; the canonical measurements (against the reference model) live

there.

Both builds are quantized structure-aware, mirroring the MLX artifact's native

allocation: the always-active backbone (attention, linear attention,

hyper-connections, shared experts) at Q6_K, embeddings and output at Q8_0, the

pruned routed experts on a compact carrier (Q3_K, or Q4_K in the UD-Q4K file),

the n-gram embedding table at Q2_K (Q3_K in UD-Q4K), and the distilled output

maps at Q8_0.

Model dimensions

| | |

|---|---|

| total / active parameters | 99B / ~5.4B |

| layers / routed experts / top-k | 48 / 512 / 10 (routed experts on 24 of the 48 layers) |

| expert intermediate size | 512 (448 used; zero-padded for the quantization block size) |

| context | as base model |

| modality | text only (vision tower not included; MTP draft head not included) |

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-gguf Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-UD-Q3K.gguf

On an M4 Max (Metal) the UD-Q3K file decodes at ~30 tokens/s.

Why a fork

The fork adds three small, general features to llama.cpp's qwen4exp

architecture, needed to store this model at its real size:

• routed banks may be absent on a layer (the router and shared expert

stay); stock llama.cpp needs uniform expert tensors, which would add ~17 GB

of all-zero experts here;

• a packed layout for the per-layer n-gram embedding table (ple.row_pack),

so block-256 quantization types apply to its 160-wide rows; otherwise the

table alone is 28.8 GB at 4.5 bits;

• an optional per-layer output map (ffn_out_map), so the distilled healing

maps ship as tensors instead of being folded into 3-bit weights.

Files produced by the fork's converter load only in the fork; upstreaming is

planned.

Notes and limitations

• Quality follows the MLX artifact: ~97% of the reference's task average and

1.56× its perplexity under the MLX protocol. Fluent and on-topic; numbers and

niche technical facts are less reliable than the reference.

• Evaluated text-only on English-web-heavy data; the base model's biases are

inherited.

• Canonical benchmarks, dimensions and the MLX-native artifact:

Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx.

Base model by the Qwen team (see license);

pruning, distillation and GGUF builds by the Niwaki project, 2026-09.