---

license: other

license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE

base_model: neopolita/Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- moe

- pruning

- quantization

- mixture-of-experts

---

!Niwaki

Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-gguf

**GGUF builds of Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx —

Qwen3.8-Flash-Next pruned to 119B total / ~5.6B active parameters, with the

program-state behaviours we test (variable tracking, two-step arithmetic,

multi-hop lookups) at the unpruned model's level — for llama.cpp on CUDA,

Vulkan, Metal and CPU. Requires the

neopolita-llama.cpp fork

(see Usage): stock llama.cpp cannot represent this model's layout yet.**

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so

that every branch serves the form of the whole. This model applies that

spirit to a Mixture-of-Experts. **A paper with the full method and

measurements is coming soon.**

Project page, with the releases and their measurements side by side:

niwakiai.com

Files

| file | size | wt2 ppl (llama.cpp, 512-ctx) |

|---|---|---|

| Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf (recommended) | 51.0 GB | 7.04 ±0.05 |

| Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q4K.gguf | 64.3 GB | 6.88 ±0.04 |

The first generation's GGUF builds under the identical protocol:

| model (UD-Q3K builds) | size | wt2 ppl |

|---|---|---|

| Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit | 49.1 GB | 7.90 ±0.05 |

| Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit | 42.5 GB | 8.79 ±0.06 |

This build reads 11% lower perplexity than the first generation's 113B

build for 1.9 GB more (the UD-Q4K file: 6.88 against 7.70).

Protocol: llama-perplexity, WikiText-2 test, 512-token windows. These

llama.cpp numbers are not directly comparable to the MLX repo's 2048-window

benchmarks; the canonical measurements (against the reference model: task

suite, KL divergence, program-state behaviours, generation battery) live

there.

Program-state check through llama.cpp (UD-Q3K file, the suite's 300

variable-tracking items, greedy decoding through llama-server): tracking

1.00 / 0.96, tracking with distractor statements 0.93 — the MLX

artifact scores 0.98 / 0.95 / 0.93 on the same items.

Both builds are quantized structure-aware, mirroring the MLX artifact's native

allocation: the always-active backbone (attention, linear attention,

hyper-connections, shared experts) at Q6_K (Q8_0 where a row is narrower than

a 256-block), embeddings and output at Q8_0, the routed experts on a compact

carrier — gate/up projections Q3_K and down projections IQ4_NL (Q4_K and Q5_0

in the UD-Q4K file; the 640-wide down rows take a 32-block type) — and the

n-gram embedding table at Q2_K (Q3_K in UD-Q4K).

Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated):

UD-Q3K 557 tok/s prompt, 30 tok/s decode; UD-Q4K 561 / 31 tok/s.

Model dimensions

| | |

|---|---|

| total / active parameters | 119B / ~5.6B |

| layers / routed experts / top-k | 48 / 512 / 10 (24 layers keep all 512 experts; 24 layers keep 16 each, and their routed path contributes only when the router picks one of them) |

| expert intermediate size | 640 (unchanged) |

| context | as base model |

| modality | text only (vision tower not included; MTP draft head not included) |

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-gguf Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-UD-Q3K.gguf

Support for this file layout landed in the fork together with this release:

if you built the fork for the first-generation files, git pull and rebuild.

Why a fork

These files use two small, general additions to llama.cpp's qwen4exp

architecture, needed to store this model at its real size:

• a layer may store a few of its routed experts under the full router

(ffn_exp_ids): 24 layers keep 16 of their 512 experts. The router still

scores all 512 and the top-10 is taken exactly as in the base model; an

expert that is not stored contributes nothing. Stock llama.cpp needs uniform

expert tensors, which would add ~28 GB of all-zero experts here;

• a packed layout for the per-layer n-gram embedding table (ple.row_pack),

so block-256 quantization types apply to its 160-wide rows; otherwise the

table alone is 28.8 GB at 4.5 bits.

Files produced by the fork's converter load only in the fork; upstreaming is

planned.

Notes and limitations

• Quality follows the MLX artifact: the reference's task average (100.5%) and

program-state behaviours at 1.26× its perplexity under the MLX protocol.

Fluent and on-topic; numbers and niche technical facts are less reliable

than the reference.

• With thinking on, the model reasons at length; use a no-thinking chat

template setting for short-form tasks.

• Evaluated text-only on English-web-heavy data; the base model's biases are

inherited.

• Canonical benchmarks, dimensions and the MLX-native artifact:

Qwen3.8-Flash-Next-119B-A5B-Niwaki-v2.4-3bit-mlx.

First generation: 113B ·

99B.

Base model by the Qwen team (see license);

pruning, recovery training and GGUF builds by the Niwaki project, 2026-09.