---

license: apache-2.0

base_model: Qwen/Qwen3-30B-A3B-Instruct-2507

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- rk3588

- rknpu2

- rockchip

- npu

- llama.cpp

- moe

- edge

language:

- en

---

Qwen3-30B-A3B-Instruct-2507 — NPU-aware GGUF for Rockchip RK3588

A GGUF quantisation of Qwen3-30B-A3B-Instruct-2507 built specifically for the

RK3588 NPU, using the RKNPU2 backend in rk-llama.cpp.

Runs comfortably on a 16 GB Radxa ROCK 5B+: 18.7 tok/s prefill, 4.9 tok/s decode

on an 1855-token prompt — faster than any public quant we tested on this hardware, in a

smaller file.

| | |

|---|---|

| file | Q30-npuaware.gguf |

| size | 13.72 GiB (3.86 bits/weight) |

| source | Qwen3-30B-A3B-Instruct-2507 Q8_0 |

| runtime | rk-llama.cpp with the RKNPU2 backend (rknpu 0.9.8 / librknnrt 2.3.x) |

---

Why a special quant for this chip

The RKNPU2 backend places a weight on the NPU only if its GGUF type is one of

F16, Q8_0, Q6_K, Q4_0. Everything else (Q3_K, Q4_K, Q5_K, IQ*) runs

on the CPU. RK3588's matmul also requires symmetric A/B precision — there is no

W4A8 — so any NPU-resident weight is read as int8 regardless of how it is stored.

That produces a sharp trade-off:

• A tensor you want the NPU to accelerate — which helps prefill, since prefill is

compute-bound — must be one of those four types.

• A tensor you want read cheaply — which helps decode, since decode is

byte-bound — must be a non-eligible type, so it stays on the CPU and is read at its

real width instead of being inflated to int8.

Prefill and decode want opposite placement. A quant choice on this chip is therefore

a placement choice, and general-purpose quants land on one side of the line by

coincidence:

| public quant | placement | outcome on RK3588 |

|---|---|---|

| Q4_0 | all eligible → NPU | good prefill; attention upcast to int8, so decode pays ~2× the bytes |

| Q3_K_M | none eligible → all CPU | good decode; NPU barely used and prefill drops sharply |

This build assigns each group to the engine that suits it.

What is inside

| tensors | type | runs on | reason |

|---|---|---|---|

| attn_q, attn_k, attn_v, attn_output (192) | Q4_0 | NPU | eligible, so the NPU accelerates prefill |

| output.weight, token_embd | Q4_0 | NPU | eligible |

| ffn_up_exps, ffn_gate_exps | Q3_K | CPU | non-eligible → read at real width; experts run on CPU under --cpu-moe anyway |

| ffn_down_exps | Q4_K | CPU | kept higher by llama.cpp's mixture heuristic, which protects the sensitive down-projection |

| norms, router | F32 | CPU | unchanged |

Type mix: 241 × F32 · 194 × Q4_0 · 96 × Q3_K · 45 × Q4_K · 3 × Q5_K.

Performance

Radxa ROCK 5B+ · RK3588 · 16 GB LPDDR5 · 4× Cortex-A76 @ 2352 MHz (pinned,

performance governor) · NVMe. 1855-token prompt, 150 output tokens, prompt caching

disabled, warm.

| quant | size | prefill tok/s | decode tok/s | turn time |

|---|---|---|---|---|

| this build | 13.72 GiB | 18.7 | 4.9 | 130 s |

| Q4_0 | 16.04 GiB | 17.8 | 4.2 | 140 s |

| Q3_K_M | 13.70 GiB | 13.4 | 5.8 | 165 s |

Against Q4_0 it improves every axis — prefill +5 %, decode +16 %, turn time −7 % — and

is 2.3 GiB smaller. Against Q3_K_M it gains +40 % prefill and −21 % turn time

for 16 % less decode; the crossover is around output ≈ 0.6 × prompt tokens, so

Q3_K_M only leads for short prompts with very long answers.

Multi-turn behaviour is strong: a follow-up question re-processes only 21 of 553

context tokens, giving 2.0 s to first token on turn two.

Usage

Needs a rk-llama.cpp build with the RKNPU2 backend.

export RKNPU_HYBRID=W8A8_STANDARD RKNPU_GLUE=1
ulimit -n 1000000

taskset -c 4-7 ./llama-server \
  -m Q30-npuaware.gguf \
  -ngl 99 --cpu-moe --no-repack \
  -np 1 -t 4 --jinja --no-warmup \
  -c 16384 -fa off -cram 2048 \
  --host 0.0.0.0 --port 8095

A few flags are worth explaining:

• RKNPU_HYBRID=W8A8_STANDARD — selects the int8 pipeline. Without it, Q4_0

weights default to the int4 W4A4_HADAMARD path, which is both slower on expert

matmuls and noticeably worse in quality.

• ulimit -n 1000000 — required. Every matmul context imports a DMA file

descriptor; the default 1024 limit is exhausted and the process crashes.

• --cpu-moe --no-repack — keeps expert weights memory-mapped as reclaimable page

cache, which is what allows a model larger than board RAM to run. --repack is on by

default and will exhaust memory on a large MoE.

• taskset -c 4-7 -t 4 — the four Cortex-A76 cores only. Including the A55 cores

costs about half the throughput; oversubscribing the A76s costs about a quarter.

• -fa off — at longer contexts the decomposed attention path outperforms ggml's

fused CPU kernel here: about +22 % decode at 1574 tokens, for ~1.5 % less prefill.

• -cram 2048 — bounds the prompt cache so it cannot evict the expert pages this

model depends on.

Optional: fewer active experts

Adding --override-kv qwen3moe.expert_used_count=int:4 routes 4 experts per token

instead of the trained 8, which gives roughly +30 % decode. It is a genuine quality

trade (measurably higher perplexity), so it is left out of the command above — enable it

only if throughput matters more than fidelity for your use case.

Reproducing

With llama-quantize, from the Q8_0:

llama-quantize --allow-requantize \
  --tensor-type attn_q=q4_0 --tensor-type attn_k=q4_0 \
  --tensor-type attn_v=q4_0 --tensor-type attn_output=q4_0 \
  --output-tensor-type q4_0 --token-embedding-type q4_0 \
  Qwen3-30B-A3B-Instruct-2507-Q8_0.gguf Q30-npuaware.gguf Q3_K_M 8

About 7 minutes on the board itself. Note that --tensor-type is a request rather than

a command — llama.cpp's k-quant mixture heuristic may promote some tensors to a higher

type — so it is worth dumping the tensor types of the result to confirm what you got.

Notes

• Tuned for a 16 GB RK3588. On a 32 GB board more of the model stays resident and

the balance shifts.

• This is deliberately not a general-purpose quant. The type choices only pay off given

the NPU eligibility rules above; on a GPU they would cost quality for no benefit.

• On this hardware, throughput is the metric to optimise, not NPU utilisation. Prefill

runs with the NPU at low occupancy while the CPU is saturated, yet moving attention

off the NPU still costs around 37 % of prefill speed.

Thanks and credits

Sincere thanks to the Qwen team at Alibaba for releasing

Qwen3-30B-A3B-Instruct-2507

under Apache-2.0 — an outstanding model, and the openness that makes work like this

possible.

• llama.cpp — ggml-org and its contributors (MIT)
• RKNPU2 ggml backend — invisiofficial/rk-llama.cpp,

which made RK3588 NPU inference possible in the first place

• Rockchip for the RKNN runtime, and Radxa for the ROCK 5B+

Quantisation and RK3588 tuning: Mojo24x7/rk-llama.cpp