---

license: apache-2.0

license_link: https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE

base_model: neopolita/Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-mlx

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- moe

- pruning

- quantization

- mixture-of-experts

---

!Niwaki

Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-gguf

**GGUF builds of Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-mlx —

Qwen3.6-35B-A3B pruned to 22.6B total / ~3.8B active parameters, with the

model's program-state tracking and few-shot reasoning intact — for llama.cpp

on CUDA, Vulkan, Metal and CPU. Requires the

neopolita-llama.cpp fork

(see Usage): stock llama.cpp cannot represent this model's layout yet.**

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so

that every branch serves the form of the whole. This model applies that

spirit to a Mixture-of-Experts. **A paper with the full method and

measurements is coming soon.**

Files

| file | size | wt2 ppl (llama.cpp, 512-ctx) |

|---|---|---|

| Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-UD-Q3K.gguf (recommended) | 11.2 GB | 8.61 ±0.06 |

| Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-Q4_K_M.gguf | 14.0 GB | 8.53 ±0.06 |

The earlier generations' GGUF builds under the identical protocol:

| model (UD-Q3K builds) | size | wt2 ppl |

|---|---|---|

| Qwen3.6-23B-A3B-Niwaki-v2.1-4bit | 11.2 GB | 9.16 ±0.06 |

| Qwen3.6-19B-A3B-Niwaki-v2-4bit | 12.2 GB | 10.05 ±0.07 |

| Qwen3.6-27B-A3B-Niwaki-2bit (first generation) | 13.5 GB | 10.41 ±0.07 |

This build beats the v2.1-4bit UD-Q3K build (9.16) by 6% at the same

size, and the first-generation flagship by 17% at 2.3 GB fewer bytes.

Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 under the identical

protocol (llama-perplexity, WikiText-2 test, 512-token windows). These

llama.cpp numbers are not directly comparable to the MLX repo's 2048-window

benchmarks; the relative standings match across both, and the canonical

measurements (task suites, gsm8k, the program-state battery) live there.

**Generation battery (measured on the canonical MLX weights; reference

scores 0.63 / 0.51 under the identical battery):** bigram-diversity avg/min

= 0.60 / 0.47 across an 8-prompt code/reasoning/chat/creative battery.

The recommended UD-Q3K build is quantized structure-aware (importance

matrices calibrated on a mixed web/code/chat/reasoning corpus), mirroring

the artifact's native allocation: the always-active backbone (attention,

shared experts, embeddings) is kept at high precision (Q6_K), the untouched

layers' routed experts ride a compact carrier (q3_k, imatrix-guided), and the

64-expert banks of layers 10–29 — the part the MLX artifact never takes

below 4-bit either — stay at q4_k. It stays within 1% of uniform Q4_K_M

quality at ~20% fewer bytes; pick Q4_K_M if you want the last percent of

quality over size.

Format note: no zero-valued pad: layers 10–29 store exactly their 64

kept experts (the fork reads a per-layer expert count), so the file carries

only experts that exist.

Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated):

UD-Q3K 1,430 tok/s prompt, 82 tok/s decode; Q4_K_M 1,475 / 91 tok/s — the

Q4_K kernels are faster than Q3_K on Metal, so Q4_K_M is the quicker file

as well as the slightly better one.

Against the intact model at the same size

The alternative to pruning is to quantize the intact 35B model harder.

Unsloth's dynamic quants

of Qwen3.6-35B-A3B that bracket these files, measured on the same machine,

build and protocol (llama-perplexity, 512-token windows, WikiText-2 test and

a 512-chunk C4 slice; llama-bench 512-token prompt / 128 generated, Metal):

| file | size | wt2 ppl ↓ | C4 ppl ↓ | prompt tok/s ↑ | decode tok/s ↑ |

|---|---|---|---|---|---|

| Qwen3.6-35B-A3B-UD-IQ2_XXS (intact) | 10.8 GB | 7.69 | 11.96 | 832 | 45 |

| this repo, UD-Q3K | 11.2 GB | 8.61 | 14.01 | 1,430 | 82 |

| Qwen3.6-35B-A3B-UD-IQ2_M (intact) | 11.5 GB | 7.59 | 11.83 | 808 | 45 |

| Qwen3.6-35B-A3B-UD-Q2_K_XL (intact) | 12.3 GB | 7.48 | 11.67 | 756 | 45 |

| Qwen3.6-35B-A3B-UD-IQ3_XXS (intact) | 13.2 GB | 7.37 | 11.44 | 796 | 41 |

| Qwen3.6-35B-A3B-UD-IQ3_S (intact) | 13.7 GB | 7.39 | 11.43 | 785 | 43 |

| this repo, Q4_K_M | 14.0 GB | 8.53 | 13.90 | 1,475 | 91 |

At equal size the intact model's dynamic quants read 11–14% lower

WikiText-2 and 15–18% lower C4 perplexity than these builds; these builds

run 1.7–2.1× faster on Metal, on prompt processing and decode alike (the

Q3_K / Q4_K kernels against the IQ kernels, and fewer routed-expert bytes

per token). Pick by what you are short of: quality at equal memory favours

the intact quants, throughput at equal memory favours these files. The

function-level comparison (task suites, gsm8k, program-state battery) is on

the MLX card.

Model dimensions

| | |

|---|---|

| total / active parameters | 22.6B / ~3.8B (8 routed + 1 shared expert per layer, unchanged) |

| layers / routed experts / top-k | 40 / 256 / 8 (layers 10–29 keep 64 experts each and route only over them) |

| expert intermediate size | 512 (unchanged) |

| context | as base model |

| conversion note | text only; speculative-decoding (MTP) draft block not included |

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-gguf Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-UD-Q3K.gguf

Why a fork

Layers 10–29 of this model keep 64 of their 256 routed experts and route

only over them. Stock llama.cpp requires every MoE layer of a model to carry

the same number of experts; the fork lets a qwen35moe layer's expert count

follow its tensors, with a router of one row per kept expert (its softmax is

exactly the renormalised routing of the MLX artifact). Files produced by the

fork's converter load only in the fork; upstreaming is planned.

Canonical benchmarks, the progress table since the first generation, and

the MLX-native artifact:

Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-mlx. Family:

v2.2-2bit ·

v2.1 4-bit / 2-bit ·

v2 4-bit / 2-bit ·

first-generation 27B-2bit · 19B-2bit · 11B-4bit.

Base model by the Qwen team (Apache 2.0); pruning, distillation, and GGUF

builds by the Niwaki project, 2026-09.