---

license: apache-2.0

base_model: Qwen/Qwen3.6-35B-A3B

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- rk3588

- rknpu2

- rockchip

- npu

- llama.cpp

- moe

- hybrid

- gated-deltanet

- edge

language:

- en

---

Qwen3.6-35B-A3B — NPU-aware GGUF for Rockchip RK3588

A GGUF quantisation of Qwen3.6-35B-A3B built specifically for the RK3588 NPU,

using the RKNPU2 backend in rk-llama.cpp.

Runs on a 16 GB Radxa ROCK 5B+: 20.4 tok/s prefill, 4.8 tok/s decode on a

1967-token prompt — the fastest prefill we have measured on this hardware, and ahead of

the public quant on every axis while being 2.1 GiB smaller.

| | |

|---|---|

| file | Q35-npuaware.gguf |

| size | 14.89 GiB (3.60 bits/weight) |

| source | bartowski/Qwen_Qwen3.6-35B-A3B-GGUF Q4_0 |

| runtime | rk-llama.cpp with the RKNPU2 backend (rknpu 0.9.8 / librknnrt 2.3.x) |

| architecture | hybrid — 30 Gated-DeltaNet/SSM blocks + 10 full-attention + 1 MTP |

---

Why a special quant for this chip

The RKNPU2 backend places a weight on the NPU only if its GGUF type is one of

F16, Q8_0, Q6_K, Q4_0. Everything else runs on the CPU. RK3588's matmul also

requires symmetric A/B precision — there is no W4A8 — so any NPU-resident weight is

read as int8 regardless of how it is stored.

So a tensor you want the NPU to accelerate (prefill, compute-bound) must be one of

those four types, while a tensor you want read cheaply (decode, byte-bound) must be a

non-eligible type so it stays on CPU at its real width. **Prefill and decode want

opposite placement**, and a quant choice becomes a placement choice.

What is inside

Only the expert tensors were changed. Every attention and SSM projection is copied

bit-exact from the source, so all of them remain NPU-eligible and the NPU continues

to carry prefill.

| tensors | count | type | runs on |

|---|---|---|---|

| ffn_up_exps, ffn_gate_exps, ffn_down_exps | 123 | Q3_K — the only change | CPU |

| attn_qkv, attn_gate (the 30 recurrent blocks) | 60 | Q4_0, unchanged | NPU |

| attn_k, attn_v | 22 | Q8_0, unchanged | NPU |

| attn_output | 11 | Q6_K | NPU |

| ssm_out | 30 | Q4_0 / Q8_0, unchanged | NPU |

| ffn_*_shexp (shared experts, read every token) | 63 | Q8_0, unchanged | NPU |

| ssm_*, norms, routers | — | F32, unchanged | CPU |

Type mix: 368 × F32 · 123 × Q3_K · 101 × Q8_0 · 83 × Q4_0 · 16 × Q6_K · 2 × BF16.

Performance

Radxa ROCK 5B+ · RK3588 · 16 GB LPDDR5 · 4× Cortex-A76 @ 2352 MHz (pinned,

performance governor) · NVMe. 1967-token prompt, 150 output tokens, prompt caching

disabled, warm.

| quant | size | prefill tok/s | decode tok/s | turn time |

|---|---|---|---|---|

| this build | 14.89 GiB | 20.4 | 4.8 | 128 s |

| UD-IQ4_XS | 16.96 GiB | 15.3 | 4.0 | 166 s |

Better on every axis: prefill +33 %, decode +20 %, turn time −23 %, in a file

2.1 GiB smaller. A live 2477-token request on a cold start measured 21.5 tok/s

prefill.

Choosing this model: long prompts, not long conversations

30 of the 41 blocks are Gated-DeltaNet. Recurrent state is a running summary, so it

cannot be rewound to an arbitrary position the way a KV cache can. A follow-up question

in the same conversation therefore re-processes most of the thread — around **94 % of the

context**, roughly 20 s to first token on turn two, growing with thread length.

That is inherent to the architecture rather than a property of this quantisation. It

makes the model an excellent fit for long single prompts — documents, code, logs,

retrieved context — and a poor fit for long back-and-forth chat, where a

non-recurrent model will feel far more responsive.

Usage

Needs a rk-llama.cpp build with the RKNPU2 backend.

export RKNPU_HYBRID=W8A8_STANDARD RKNPU_GLUE=1
export LLAMA_RECURRENT_ON_CPU=1
ulimit -n 1000000

taskset -c 4-7 ./llama-server \
  -m Q35-npuaware.gguf \
  -ngl 99 --cpu-moe --no-repack \
  -np 1 -t 4 --jinja --no-warmup \
  -c 32768 -fa on -cram 2048 \
  --host 0.0.0.0 --port 8095

• LLAMA_RECURRENT_ON_CPU=1 is essential for this model. It assigns the recurrent

layers to the CPU device, which is what allows the fused Gated-DeltaNet kernels to

stay enabled. Without it both fused paths are disabled silently — no error, just a

large slowdown. Check the load log for

fused Gated Delta Net … enabled; it is the most important line for this

architecture. (This variable is provided by the fork linked below.)

• RKNPU_HYBRID=W8A8_STANDARD — selects the int8 pipeline; without it eligible

weights fall to the int4 path, which is slower and worse.

• -fa on here, unlike on a pure-attention model: only 10 of 41 blocks attend, so

the fused kernel is roughly neutral and on edges ahead.

• -c 32768 is inexpensive on this architecture — KV costs about 20 KiB/token,

versus roughly 96 KiB/token on a comparable dense-attention MoE.

• --cpu-moe --no-repack — keeps experts memory-mapped as reclaimable page cache,

which is what lets a model larger than board RAM run. --repack is on by default and

will exhaust memory on a large MoE.

• ulimit -n 1000000 — required; each matmul context imports a DMA file descriptor

and the default 1024 limit is exhausted.

• taskset -c 4-7 -t 4 — A76 cores only; including the A55s costs about half the

throughput.

Optional: fewer active experts

Adding --override-kv qwen35moe.expert_used_count=int:4 routes 4 of 256 experts per

token instead of the trained 8. On this model the trained top-8 setting costs roughly

44 % of decode speed, so the override is a large throughput win — but it is a real

quality trade, so it is left out of the command above.

Reproducing

Because only the experts change, a full per-tensor type file is used so every other

tensor is copied rather than requantised. llama-quantize copies a tensor when its

target type already equals its current type, so listing every tensor at its current type

makes each one a no-op:

# every tensor at its current type, except the three expert groups -> q3_K
python3 gen_ttypes.py \
  --targets=ffn_up_exps.weight,ffn_gate_exps.weight,ffn_down_exps.weight \
  Qwen_Qwen3.6-35B-A3B-Q4_0.gguf tt35.txt q3_K

llama-quantize --allow-requantize --tensor-type-file tt35.txt \
  Qwen_Qwen3.6-35B-A3B-Q4_0.gguf Q35-npuaware.gguf Q4_0 8

gen_ttypes.py is in the fork linked below. Worth checking the resulting tensor types:

llama.cpp's mixture heuristic can promote a tensor to a higher type than requested.

Notes

• Tuned for a 16 GB RK3588; on a 32 GB board the balance shifts.
• This tokenizer produces roughly 6 % more tokens for the same text than

Qwen3-30B (vocabulary 248,320 vs 151,936), which is worth remembering when comparing

tokens/s figures between the two.

• Deliberately not a general-purpose quant — the type choices only pay off given the NPU

eligibility rules above.

• Optimise for throughput rather than NPU utilisation. Prefill runs at low NPU occupancy

while the CPU is saturated, yet moving attention off the NPU still costs a large share

of prefill speed.

Thanks and credits

Sincere thanks to the Qwen team at Alibaba for releasing

Qwen3.6-35B-A3B under Apache-2.0 — a

remarkable hybrid architecture, and the kind of openness that makes edge work like this

possible.

Thanks also to bartowski, whose

Qwen_Qwen3.6-35B-A3B-GGUF

conversion was the starting point for this build.

• llama.cpp — ggml-org and its contributors (MIT)
• RKNPU2 ggml backend — invisiofficial/rk-llama.cpp,

which made RK3588 NPU inference possible in the first place

• Rockchip for the RKNN runtime, and Radxa for the ROCK 5B+

Quantisation and RK3588 tuning: Mojo24x7/rk-llama.cpp