---

license: other

license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE

base_model: neopolita/Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-mlx

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- moe

- pruning

- quantization

- mixture-of-experts

---

!Niwaki

Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-gguf

**GGUF builds of Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-mlx —

Qwen3.8-Flash-Next pruned to 102B total / ~5.3B active parameters, with the

variable-tracking behaviours the first-generation 99B lost back at the

unpruned model's level — for llama.cpp on CUDA, Vulkan, Metal and CPU.

Requires the

neopolita-llama.cpp fork

(see Usage): stock llama.cpp cannot represent this model's layout yet.**

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so

that every branch serves the form of the whole. This model applies that

spirit to a Mixture-of-Experts. **A paper with the full method and

measurements is coming soon.**

Project page, with the releases and their measurements side by side:

niwakiai.com

Files

| file | size | wt2 ppl (llama.cpp, 512-ctx) |

|---|---|---|

| Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf (recommended) | 43.9 GB | 8.75 ±0.06 |

| Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q4K.gguf | 55.9 GB | 8.57 ±0.06 |

The first generation's GGUF builds under the identical protocol:

| model (UD-Q3K builds) | size | wt2 ppl |

|---|---|---|

| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit | 42.5 GB | 8.79 ±0.06 |

| Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit | 49.1 GB | 7.90 ±0.05 |

Perplexity is level with the first-generation 99B build; what this generation

adds is function — see the program-state table on the MLX card, where the

tracking families go from 0.58–0.75 to 0.95–0.96. The larger sibling,

119B-A5B v2.4,

reads 7.04 at 51.0 GB.

Protocol: llama-perplexity, WikiText-2 test, 512-token windows. These

llama.cpp numbers are not directly comparable to the MLX repo's 2048-window

benchmarks; the canonical measurements (against the reference model: task

suite, KL divergence, program-state behaviours, generation battery) live

there.

Program-state check through llama.cpp (UD-Q3K file, the suite's 300

variable-tracking items, greedy decoding through llama-server): tracking

0.96 / 0.93, tracking with distractor statements 0.92 — the MLX

artifact scores 0.98 / 0.93 / 0.90 on the same items.

Both builds are quantized structure-aware, mirroring the MLX artifact's native

allocation: the always-active backbone (attention, linear attention,

hyper-connections, shared experts) at Q6_K (Q8_0 where a row is narrower than

a 256-block), embeddings and output at Q8_0, the routed experts on a compact

carrier — Q3_K (Q4_K in the UD-Q4K file), with the 640-wide down projections of

the 24 layers' kept experts on a 32-block type (IQ4_NL, or Q5_0 in UD-Q4K) —

and the n-gram embedding table at Q2_K (Q3_K in UD-Q4K).

Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated):

UD-Q3K 431 tok/s prompt, 26 tok/s decode; UD-Q4K 430 / 27 tok/s.

Model dimensions

| | |

|---|---|

| total / active parameters | 102B / ~5.3B |

| layers / routed experts / top-k | 48 / 512 / 10 (24 layers keep all 512 experts, narrowed; 24 layers keep 26–31 full-width experts each, and their routed path contributes only when the router picks one of them) |

| expert intermediate size | 512 on the 24 full layers (448 used; zero-padded for the quantization block size); 640 for the kept experts of the other 24 |

| context | as base model |

| modality | text only (vision tower not included; MTP draft head not included) |

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-gguf Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf

Support for this file layout landed in the fork together with this release:

if you built the fork earlier, git pull and rebuild.

Why a fork

These files use three small, general additions to llama.cpp's qwen4exp

architecture, needed to store this model at its real size:

• a layer may store a few of its routed experts under the full router

(ffn_exp_ids): 24 layers keep 26–31 of their 512 experts. The router still

scores all 512 and the top-10 is taken exactly as in the base model; an

expert that is not stored contributes nothing. Stock llama.cpp needs uniform

expert tensors, which would add ~27 GB of all-zero experts here;

• each layer's expert width comes from its tensors: the kept experts are

640 wide, every other layer's experts 512;

• a packed layout for the per-layer n-gram embedding table (ple.row_pack),

so block-256 quantization types apply to its 160-wide rows; otherwise the

table alone is 28.8 GB at 4.5 bits.

Files produced by the fork's converter load only in the fork; upstreaming is

planned.

Notes and limitations

• On Apple Silicon, llama.cpp keeps this whole file in GPU memory, its

n-gram embedding table included: about 40.8 GiB for UD-Q3K and 52.1 GiB

for UD-Q4K, plus the context. Our newer GGUFs keep the table out of GPU

memory: Niwaki-Flash-Next-23GB-v4.0-gguf needs

21.8 GiB (wt2 7.16) and

Niwaki-Qwen3.8-Flash-Next-141B-A6B-v4.0-gguf

42.0 GiB (wt2 5.76).

• Quality follows the MLX artifact: ~97% of the reference's task average at

1.55× its perplexity under the MLX protocol, with variable tracking at the

reference's level and two-step arithmetic just below it (0.87 against

0.91). Fluent and on-topic; numbers and niche technical facts are less

reliable than the reference.

• With thinking on, the model reasons at length; use a no-thinking chat

template setting for short-form tasks.

• Evaluated text-only on English-web-heavy data; the base model's biases are

inherited.

• Canonical benchmarks, dimensions and the MLX-native artifact:

Qwen3.8-Flash-Next-102B-A5B-Niwaki-v2.4-3bit-mlx.

First generation: 99B ·

113B.

Base model by the Qwen team (see license);

pruning, recovery training and GGUF builds by the Niwaki project, 2026-09.