---

license: other

license_name: qwen-community-1.0

base_model:

• Qwen/Qwen3.8-Flash-Next
• unsloth/Qwen3.8-Flash-Next-GGUF
• ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF

tags:

• gguf
• mtp
• speculative-decoding
• llama.cpp

---

Qwen3.8-Flash-Next — separate MTP head on the expert-pruned (256-expert) GSQ-RCO model

Field notes from getting --spec-type draft-mtp to run on

ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF (176B MoE, pruned 512 → 256 experts)

with a separate MTP head file, plus a smaller, expert-pruned variant of that head.

Short version

1. The unmodified unsloth head (mtp-Qwen3.8-Flash-Next-Q8_0.gguf, 512 experts) works as-is

on the 256-expert model. The head is loaded as its own model with its own expert_count;

the mismatch does not matter.

1. What you need is a llama.cpp build that contains "Qwen4Exp: add MTP" (#29761, 2026-10-01)

plus the one-hunk patch in this repo. Without the patch, -md <head-only file> aborts

with GGML_ASSERT(buffer) failed during server start.

1. On CPU the head is a clear win. On GPU, in my setup, it helps code and hurts prose, and costs context (numbers below).
2. The file in this repo is the same head with its draft layer cut to 256 experts: 2.8 GB

instead of 4.1 GB, at a lower acceptance rate.

Update 2026-10-18: what I actually run now (MTP on, GPU)

The verdict further down ("MTP off on GPU") is outdated. The fixed multi-token surcharge

described below turned out to be a llama.cpp kernel issue, and once that was fixed the

original 512-expert head became a clear win on GPU as well.

Cause of the surcharge: about 1.5 GB of thin BF16 matrices (hyper-connections hc_*

10240×320, routers, indexer). On Ampere, mmvf.cu only uses the vector kernel for BF16 when

ne11 == 1; from 2 tokens per step on it falls back to cuBLAS/MMF, and BF16 time goes from

4.2 to 15.9 ms per step (seen in nsys). Upstream #29633 fixes this.

Running setup (2× RTX 3080 20 GB, owl model ID 121):

• llama.cpp 46ca246 + cherry-picked #29633 (BF16 multi-token kernel) + #29825

(smaller compute buffers) + the one-hunk patch from this repo. Current master crashes with a

head plus parallel slots (ops.cpp:5240 GGML_ASSERT(i01 >= 0 && i01 < ne01) in get_rows_q),

so I stay on the cherry-picked build.

• Head: the original 512-expert head, requantised to Q4_K_M

(llama-quantize --allow-requantize from the Q8_0 head, 2.6 GB). Not the pruned head from

this repo.

• --spec-type draft-mtp --spec-draft-n-max 2 --parallel 2 --ctx-size 196608 (2×96K),

--tensor-split 45,55 -devd CUDA0 -ctkd q5_1 -ctvd q5_1, main KV q8_0,

--no-mmproj-offload (vision projector on the GPU runs card 0 out of memory under load),

--ctx-checkpoints 8, --cache-ram 32768.

Heads compared, experts against bits (server, 1 slot, n-max 2, 3 runs each, temperature 0):

| head | size | code tok/s (acceptance) | prose tok/s (acceptance) |

|---|---|---|---|

| none | – | 61.6 | 61.4 |

| 512 experts Q8_0 | 3.9 GB | 94.2 (89 %) | 68.6 (49 %) |

| 512 experts Q6_K | 3.2 GB | 96.6 (87 %) | 70.8 (49 %) |

| 512 experts Q4_K_M (in use) | 2.6 GB | 99.4 (91 %) | 70.3 (47 %) |

| 256 experts Q8_0 (this repo) | 2.7 GB | 92.7 (87 %) | 64.7 (43 %) |

| 256 experts Q6_K | 2.2 GB | 94.9 (86 %) | 66.2 (43 %) |

| 256 experts Q4_K_M | 1.8 GB | 97.1 (89 %) | 68.1 (45 %) |

The number of experts matters, the bits hardly do. Within one head size the acceptance rates

are within noise, but the 512 head accepts 4–6 points more on prose. A smaller quantised head

is even slightly faster because drafting gets cheaper. The 512 Q4_K_M is the size of the 256

Q8_0 and better in every column.

Re-check today (2026-10-18), production setup above, 2×96K, single request:

code 93–98 tok/s (88 % acceptance), German prose 77 tok/s (~60 %). Tool calls and reasoning work.

Limits found:

• 2×96K holds with two sessions filled to 88K at the same time. Speed drops with context:

~62 tok/s at 30K, ~34 tok/s at 88K.

• 3×96K with a head does not load (compute buffer on card 1). 3×64K loads.
• 1×192K loads but dies with CUDA OOM on the first 90K prompt: compute buffers grow with the

length of a single sequence, not with total KV size.

Update 2026-10-18 (later): 2 vs 3 sessions, and the eudaimonent 256-router head measured

The question was whether a smaller head frees enough VRAM for a third session, and whether

that third session gains more total throughput than the smaller head loses.

Answer on 2× RTX 3080 20 GB: no.

Same build and flags as above, n-max 2, 400 tokens per request, temperature 0, thinking off,

2 runs per cell. "Total" = all tokens of all concurrent requests / wall-clock time.

| setup | code, 1 request | code, all slots busy | prose, 1 request | prose, all slots busy |

|---|---|---|---|---|

| A: 512 head Q4_K_M, 2×96K (in use) | 95.7 | 125.2 (2 concurrent) | 72.0 | 102.7 (2 concurrent) |

| B: eudaimonent 256 head Q4_0, 3×80K | 93.3 | 116.3 (3 concurrent) | 67.7 | 84.3 (3 concurrent) |

| C: 512 head Q4_K_M, 3×64K | 95.8 | 121.3 (3 concurrent) | 73.0 | 99.6 (3 concurrent) |

| D: eudaimonent 256 head Q4_0, 2×96K | 92.0 | 117.7 (2 concurrent) | 67.3 | 96.7 (2 concurrent) |

Findings:

• 3×96K does not work with any head on 2× 20 GB. With --tensor-split 45,55 it loads but

dies with CUDA OOM on card 1 at the second short request (main compute buffer 1.84 GB).

From 47,53 on, the head's own compute buffer (778 MB) no longer fits on card 0. -ub 256

does not help. With the small head, 3×80K was the maximum.

• --parallel 3 costs throughput even with only two requests: 103.9 vs 125.2 tok/s (code)

for two concurrent requests, compared with --parallel 2. Three busy slots (121) do not

beat two busy slots (125).

• The eudaimonent head is slightly worse everywhere than the 512 head at Q4_K_M: prose

acceptance about 52 % vs 60 %, code 93 % vs 98 % at one request. Its VRAM saving lands on

card 0, while the bottleneck is card 1.

• Its 256 experts overlap with the frequency-based selection in this repo in only 124 of 256

(random would be ~128), so it is a genuinely different selection, not the same one.

Production stays at 512 head Q4_K_M, 2×96K.

Note on the other 256-router head

Another 256-expert head for the same main model has been published:

eudaimonent/mtp-Qwen3.8-Flash-Next-Q4_0-256router

(Q4_0, 1.65 GB, re-mapped from the ggml-org 512-expert head).

Measured on 2026-10-18, see the section above. Two further points:

• Re-mapping is not required for compatibility. With a llama.cpp build that has #29761,

the head is loaded as a separate model with its own expert_count, and the unmodified

512-expert head runs on the 256-expert main model.

• In my measurements the cut-down heads always lost some acceptance (see the table above).

Their only advantage was VRAM: a 256 head fitted 3×64K under load where the 512 Q8_0 head

ran out of memory. If you are short on memory, a 256 head is a reasonable trade-off. If not,

the 512 head at Q4_K_M was the best option here.

The model card does not say which 256 experts were kept or how they were chosen, so I cannot

compare that part with the frequency heuristic used here.

Files

| file | what |

|---|---|

| mtp-Qwen3.8-Flash-Next-256exp-Q8_0.gguf | unsloth full head, blk.48 experts sliced 512 → 256, expert_count patched. 2.8 GB |

| qwen4exp-mtp-only-kpool.patch | patch against llama.cpp 46ca246 (src/models/qwen4exp.cpp) |

| expert_map.json | which 256 of the 512 experts survive in each of the 48 layers of the pruned release |

| prune_head.py | the script that produced the GGUF (no requantisation, whole Q8_0 blocks are copied) |

Usage

llama-server -m Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf \
  -md mtp-Qwen3.8-Flash-Next-256exp-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2

Only the full head variant loads this way. The shared variant

(nextn_shared_target_tensors) is rejected by upstream llama.cpp with

tensor 'token_embd.weight' not found.

Measurements

One run per cell, temperature 0, thinking off. Treat them as indications, not benchmarks.

CPU only (8 threads, ctx 4096, one code prompt, 160 tokens)

| | tok/s | draft acceptance |

|---|---|---|

| no speculation | 3.49 | – |

| original head, 512 experts | 5.61 (+61 %) | 81.7 % (98/120) |

| this head, 256 experts | 5.13 (+47 %) | 74.8 % (95/127) |

GPU (2× 20 GB, 3 slots, KV q5_1, 3 runs of 400 tokens each, same build for all rows)

Baseline without a head: 61.5 tok/s for code and prose alike. ngram-map-k adds nothing

(60.3 / 61.5).

| setting | code | prose (German) | notes |

|---|---|---|---|

| 256 head, 3×64K, n-max 1 | 55.4 | 49.4 | |

| 256 head, 3×64K, n-max 2 | 67.1 | 50.3 | |

| 256 head, 3×64K, n-max 3 | 70.5 | 59.6 (45–70) | unstable: CUDA OOM under three parallel 54k prompts |

| 256 head, 3×64K, n-max 3, p-min 0.6 | 58.9 | 41.2 | |

| 256 head, 3×48K, n-max 3 | 64.0 | 41.9 | stable under three parallel 39k prompts |

| original 512 head, 3×32K, n-max 3 | 75.5 | 54.3 | not tested under load |

| either head, 3×96K | – | – | does not load |

To make a head fit at all:

-devd CUDA0 --tensor-split 45,55 -ctkd q5_1 -ctvd q5_1

-devd puts the head on the first card and the uneven split makes room for it there;

-ctkd/-ctvd matter because the head's own KV cache is f16 otherwise. Without these the head

and its context land on one card (about 3.2 GB) and only 3×32K with the pruned head loads.

Verdict for this hardware (2026-10-02, superseded, see the update above): code +4 to +23 %, prose −3 to −32 %, and the usable context

drops from 3×96K to 3×48K (256 head) or 3×32K (original head). I left MTP switched off on

GPU. On CPU it is worth it.

(An earlier version of this card had single-run numbers and said the original head does not

fit on GPU; that was before the head was moved to the other card.)

Why the gain is small: the multi-token step

Things that turned out not to matter (all re-measured, 3 runs each, new build, KV q8_0):

| setting | code | prose |

|---|---|---|

| no head, 1 slot | 61.9 | 61.6 |

| 256 head, 1 slot, n-max 2 | 68.5 (acceptance 81 %) | 52.7 (50 %) |

| original 512 head, 1 slot, n-max 2 | 69.1 (82 %) | 58.6 (61 %) |

| 256 head, 3 slots, n-max 2 | 68.6 (81 %) | 52.1 (49 %) |

| no head, 3 slots | 61.3 | 60.6 |

• -np 1 versus 3 slots: no difference.
• KV cache q8_0 versus q5_1: no difference.
• Which card the head sits on: no difference.

What does matter is the cost of one decode step of the main model as a function of how

many tokens it processes at once (measured without any head, via prompt evaluation with

-b N -ub N; the 1-token figure is from normal generation):

| tokens per step | time per step |

|---|---|

| 1 | 16.2 ms |

| 2 | 28.6 ms |

| 3 | 30.6 ms |

| 4 | 33.3 ms |

| 8 | 45.9 ms |

Going from one to two tokens costs about 12 ms extra, each further token only 2–3 ms. MTP

verification always needs such a multi-token step. With a verify step of 3 tokens plus the

head (~36 ms in total) this predicts the measurements well: 82 % acceptance gives about

2.5 tokens per step, i.e. ~70 tok/s (measured 68.5); 50 % acceptance gives ~49 tok/s

(measured 52.7). Even perfect acceptance would top out near 85 tok/s against 62 without MTP,

so retraining the head would not change the picture much.

The surcharge is a fixed amount per multi-token step. It does not depend on the allocated

context or on how full the cache is:

| context, tokens in cache | 3-token step |

|---|---|

| 16K, 450 | 30.7 ms |

| 64K, 450 | 30.8 ms |

| 262K, 450 | 30.9 ms |

| 64K, 6000 | 31.2 ms |

That rules out two explanations I had been pointed to: expert weights spilling into system

RAM (the model is fully in VRAM here) and a dense attention fallback that scans the whole

cache (it would scale with the cache). Where the fixed cost comes from: BF16 matrices falling off the vector kernel, see the update above (#29633).

The 1-token figure comes from generation and the others from prompt evaluation, so the

comparison is approximate.

Practical notes: -b 1 makes the server abort at start (the verify batch needs n-max + 1

tokens), and lowering --spec-draft-p-min to 0.6 made things worse, which fits a nearly

fixed step price. Build used: llama.cpp b11347-17-g46ca246 plus the patch in this repo.

How the pruned head was made, and its limits

expert_map.json was recovered exactly, without any calibration run: the MoE routers

(blk.N.ffn_gate_inp.weight) are stored unquantised (BF16) and are byte-identical between

the unpruned release (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF) and the pruned one.

Cosine similarity between the 256 pruned and 512 original router rows gives 1.0 and 256

unique indices in every layer. Only the router tensors were fetched, via HTTP range requests.

That map does not tell you which experts the MTP head needs:

• every layer keeps a different subset; the union over all layers is all 512 experts;
• the head's draft layer (blk.48) has its own router, which matches none of the 48 trunk

routers (best cosine similarity of the shared-expert gate: 0.12).

The head here therefore keeps the 256 experts that survive in the most trunk layers — a

heuristic. A selection based on the draft router's own usage statistics would likely do

better. The draft layer was not retrained.

Credits and license

Head weights: unsloth (unsloth/Qwen3.8-Flash-Next-GGUF), base model by Qwen, pruned and

quantised main model by ISTA-DASLab. This repo only re-slices existing weights; the

upstream license (qwen-community-1.0) applies.