---

license: apache-2.0

license_link: https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE

base_model: neopolita/Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-mlx

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- moe

- pruning

- quantization

- mixture-of-experts

---

!Niwaki

Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-gguf

**GGUF builds of Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-mlx —

Qwen3.6-35B-A3B pruned to 22.6B total / ~3.8B active parameters, with the

model's program-state tracking and few-shot reasoning intact — for llama.cpp

on CUDA, Vulkan, Metal and CPU. Requires the

neopolita-llama.cpp fork

(see Usage): stock llama.cpp cannot represent this model's layout yet.**

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so

that every branch serves the form of the whole. This model applies that

spirit to a Mixture-of-Experts. **A paper with the full method and

measurements is coming soon.**

Files

| file | size | wt2 ppl (llama.cpp, 512-ctx) |

|---|---|---|

| Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-UD-Q3K.gguf (recommended) | 11.2 GB | 9.69 ±0.07 |

| Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-Q4_K_M.gguf | 14.0 GB | 9.74 ±0.07 |

These files are the GGUF counterpart of the 2-bit MLX artifact: its experts

re-quantized onto llama.cpp's 3- and 4-bit carriers, which makes them the

same size as the sibling

Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-gguf

builds (8.61 / 8.53 under the identical protocol). At these sizes the

sibling is the better choice; pick this repo when you want the 2-bit

artifact's exact function in llama.cpp.

The earlier generations' 2-bit GGUF builds under the identical protocol:

| model (UD-Q3K builds) | size | wt2 ppl |

|---|---|---|

| Qwen3.6-23B-A3B-Niwaki-v2.1-2bit | 11.2 GB | 10.36 ±0.07 |

| Qwen3.6-19B-A3B-Niwaki-v2-2bit | 12.2 GB | 11.44 ±0.08 |

| Qwen3.6-19B-A3B-Niwaki-2bit (first generation) | 9.0 GB | 13.15 ±0.10 |

Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 under the identical

protocol (llama-perplexity, WikiText-2 test, 512-token windows). These

llama.cpp numbers are not directly comparable to the MLX repo's 2048-window

benchmarks; the relative standings match across both, and the canonical

measurements (task suites, gsm8k, the program-state battery) live there.

**Generation battery (measured on the canonical MLX weights; reference

scores 0.63 / 0.51 under the identical battery):** bigram-diversity avg/min

= 0.56 / 0.47 across an 8-prompt code/reasoning/chat/creative battery.

The recommended UD-Q3K build is quantized structure-aware (importance

matrices calibrated on a mixed web/code/chat/reasoning corpus), mirroring

the artifact's native allocation: the always-active backbone (attention,

shared experts, embeddings) is kept at high precision (Q6_K), the untouched

layers' routed experts ride a compact carrier (q3_k, imatrix-guided), and the

64-expert banks of layers 10–29 — the part the MLX artifact never takes

below 4-bit either — stay at q4_k. It matches uniform Q4_K_M quality (9.74)

at ~20% fewer bytes.

Format note: no zero-valued pad: layers 10–29 store exactly their 64

kept experts (the fork reads a per-layer expert count), so the file carries

only experts that exist.

Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated):

UD-Q3K 1,445 tok/s prompt, 82 tok/s decode; Q4_K_M 1,465 / 91 tok/s — the

Q4_K kernels are faster than Q3_K on Metal.

Against the intact model at the same size

The alternative to pruning is to quantize the intact 35B model harder.

Unsloth's dynamic quants

of Qwen3.6-35B-A3B that bracket these files, measured on the same machine,

build and protocol (llama-perplexity, 512-token windows, WikiText-2 test and

a 512-chunk C4 slice; llama-bench 512-token prompt / 128 generated, Metal):

| file | size | wt2 ppl ↓ | C4 ppl ↓ | prompt tok/s ↑ | decode tok/s ↑ |

|---|---|---|---|---|---|

| Qwen3.6-35B-A3B-UD-IQ2_XXS (intact) | 10.8 GB | 7.69 | 11.96 | 832 | 45 |

| this repo, UD-Q3K | 11.2 GB | 9.69 | 15.33 | 1,445 | 82 |

| Qwen3.6-35B-A3B-UD-IQ2_M (intact) | 11.5 GB | 7.59 | 11.83 | 808 | 45 |

| Qwen3.6-35B-A3B-UD-Q2_K_XL (intact) | 12.3 GB | 7.48 | 11.67 | 756 | 45 |

| Qwen3.6-35B-A3B-UD-IQ3_XXS (intact) | 13.2 GB | 7.37 | 11.44 | 796 | 41 |

| Qwen3.6-35B-A3B-UD-IQ3_S (intact) | 13.7 GB | 7.39 | 11.43 | 785 | 43 |

| this repo, Q4_K_M | 14.0 GB | 9.74 | 15.42 | 1,465 | 91 |

At equal size the intact model's dynamic quants read 20–24% lower

WikiText-2 and 22–26% lower C4 perplexity than these builds (the 4-bit

sibling's builds, at the same sizes, narrow that to 11–18%); these builds

run 1.7–2.1× faster on Metal, on prompt processing and decode alike (the

Q3_K / Q4_K kernels against the IQ kernels, and fewer routed-expert bytes

per token). Pick by what you are short of: quality at equal memory favours

the intact quants, throughput at equal memory favours these files. The

function-level comparison (task suites, gsm8k, program-state battery) is on

the MLX card.

Model dimensions

| | |

|---|---|

| total / active parameters | 22.6B / ~3.8B (8 routed + 1 shared expert per layer, unchanged) |

| layers / routed experts / top-k | 40 / 256 / 8 (layers 10–29 keep 64 experts each and route only over them) |

| expert intermediate size | 512 (unchanged) |

| context | as base model |

| conversion note | text only; speculative-decoding (MTP) draft block not included |

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-gguf Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-UD-Q3K.gguf

Why a fork

Layers 10–29 of this model keep 64 of their 256 routed experts and route

only over them. Stock llama.cpp requires every MoE layer of a model to carry

the same number of experts; the fork lets a qwen35moe layer's expert count

follow its tensors, with a router of one row per kept expert (its softmax is

exactly the renormalised routing of the MLX artifact). Files produced by the

fork's converter load only in the fork; upstreaming is planned.

Canonical benchmarks, the progress table since the first generation, and

the MLX-native artifact:

Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-mlx. Family:

v2.2-4bit ·

v2.1 4-bit / 2-bit ·

v2 4-bit / 2-bit ·

first-generation 27B-2bit · 19B-2bit · 11B-4bit.

Base model by the Qwen team (Apache 2.0); pruning, distillation, and GGUF

builds by the Niwaki project, 2026-09.