---

license: other

license_name: qwen-community-1.0

license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE

base_model: Qwen/Qwen3.8-Flash-Next

tags:

• gguf
• rocmfp4
• rocm
• strix-halo
• gfx1151
• quantized

---

Qwen3.8-Flash-Next — Q4_0_ROCMFP4_STRIX_LEAN

ROCmFP4 quantization of Qwen/Qwen3.8-Flash-Next (arch qwen4exp, 180B total: 125B base / 6B active + 51B PLE n-gram table + 4B MTP), tuned for AMD Strix Halo (Radeon 8060S, gfx1151) unified memory.

Run it with our engine

These tensor types need the ROCmFPX fork family — stock llama.cpp cannot load them.

Recommended runtime: our lab build (pugant/strix-halo-llamacpp-lab, main) —

full source, Dockerfile and replication guides. On top of the fork it adds:

• the qwen4exp architecture port + external MTP drafter (-md), incl. the rollback

fixes the drafter requires (qwen4exp series, patch 0012 or later);

• --ple-disk — the PLE table stays on disk: this quant fits 128 GB with ~36 GB of RAM to spare;
• --cache-disk-persist — the prompt cache survives server restarts (a 107k-token

context: 1.57 s restore, 14.3 s end-to-end vs the measured 920 s cold re-prefill —

64×; token-exact boundary with the drafter);

• vision (mmproj) together with the drafter; the reasoning-budget warn window.

Guides: PLE disk-offload · persistent prompt cache.

Recipe (ftype 106, Q4_0_ROCMFP4_STRIX_LEAN)

| Tensor group | Quant |

|---|---|

| MoE experts ffn_*_exps (48×512) | ROCmFP4 (type 101) |

| Shared experts ffn_*_shexp | ROCmFP4 |

| Attention | ROCmFP4 / ROCmFP4-FAST mix |

| PLE n-gram table per_layer_token_embd (51.2B) | Q5_1 (--tensor-type override) |

| token_embd | Q5_K |

| output (lm head) | Q6_K |

| Norms / hash constants | F32 / U64 |

• Quantized from an FP8 official checkpoint → Q8_0 intermediate → LEAN (--allow-requantize; the q8 step is loss-noise at 4-bit).
• Imatrix: our own calibration (grug-think reasoning / IT prose / code mix), 926 entries over 256 chunks, computed on a Q4_K intermediate (community practice; declared here for transparency).
• MTP draft head not included (converter drops it, as vLLM does); pair it with an external MTP drafter via -md — measured below with agentionai/Qwen3.8-Flash-Next-MTP-Q8_0-GGUF (the community Q8_0 MTP head).
• Size: 98.47 GiB (105,753,530,976 bytes), 4.78 BPW — fits the same ~64 GiB resident envelope as the reference build.

Serving (Strix Halo)

llama-server -m Qwen3.8-Flash-Next-Q4_0_ROCMFP4_STRIX_LEAN-00001-of-0000N.gguf \
  --n-gpu-layers 999 --flash-attn on --ctx-size 8192 --threads 16 --jinja

• Keep mmap ON (default): the PLE table is a sparse lookup — --no-mmap OOMs via cgroup on unified memory with no server-log evidence.
• Point -m at the first shard.

Validation

• Conversion sanity: perplexity of the Q4_K intermediate on an Italian holdout (Dante), upstream PR #27742 runtime, CPU: PPL = 1.156 ± 0.006 (64 chunks). This holdout is memorized by the base model (PPL near 1 is impossible on unseen literary text), so the number is not a quality benchmark — it is a pipeline check: broken weights (e.g. mishandled FP8 scales) would blow it up by orders of magnitude (the random-init toy of the same architecture measures ~250k).
• Cross-runtime agreement: the same holdout through the ROCmFPX fork runtime (this LEAN, ftype 106) reads PPL = 1.095 — two independent implementations, one file.

Benchmarks

On-device, Radeon 8060S (gfx1151) via ROCm 7.2.4 on the ROCmFPX fork, 999/999 layers, flash-attn on, median of 3 (llama-bench -t 16 -fa 1 -r 3):

| test | tok/s | flags |

|---|---|---|

| tg128 | 22.03 ± 0.23 | -b 2048 |

| pp2048 | 131.12 ± 21.74 | -b 2048 -ub 512 |

| tg128 | 20.41 ± 0.17 | default batch |

| pp512 | 85.99 ± 4.42 | default batch |

• Reference build by kingjones777 (same preset, no imatrix): tg 22.6, pp 345 — our tg is on par while carrying an importance-matrix calibration; pp has headroom left in batch/kernel tuning (first-cut port).
• Keep mmap ON: the PLE table is a sparse lookup, resident stays ~64 GiB at 8k ctx like the reference.

Speculative decoding — external MTP drafter (measured on-device)

Pair this target with the external MTP head agentionai/Qwen3.8-Flash-Next-MTP-Q8_0-GGUF (Q8_0, 3.85 GiB — Q8 or better; lighter quants collapse draft acceptance). Requires our lab build (main, qwen4exp series, patch 0012 or later) — see Run it with our engine above. On older builds the drafter works, but every rollback round degrades the output.

llama-server -m <this-LEAN-shard-1> -md Qwen3.8-Flash-Next-MTP-Q8_0.gguf -ngl 999 --spec-draft-ngl all -fa on --jinja -c 8192 -fit off --spec-type draft-mtp --spec-draft-n-max 5 --spec-draft-p-min 0.75 --spec-draft-p-split 0.10

Dedicated GPU, ctx 8192, temp 0, median of 3 (/v1/chat/completions, 600 max tokens), same container image for both arms:

| workload | plain | + MTP n=3 | + MTP n=5 | Δ (n=3 / n=5) |

|---|---|---|---|---|

| deterministic (counting 1-200) | 22.1 | 46.0 | 50.2 | +108% / +127% |

| deterministic (alphabet) | 20.9 | 32.0 | 32.6 | +53% / +56% |

| open prose (Italian) | 22.6 | 22.8-25.4 | — | +1% to +12% |

• Draft acceptance: 95.7% of tokens, mean accepted length 3.24 at n-max 3 (per-position 1.00 / 0.72 / 0.53); on deterministic text it stays high out to position 5 (5.20/6) and throughput still climbs with n.
• Backend matrix: the port runs unchanged on HIP (ROCm 7.2.4) and Vulkan (RADV) — Vulkan plain is +4-11% vs HIP; with MTP the two backends converge (Vulkan MTP boost +9% to +95%).
• Where the money is: the round bottleneck is the batched verify on the hybrid trunk (KV + GDN scan + QSA index + PLE lookup), not the drafter (~16 ms/round, ~5% of wall). Deterministic work converts acceptance into speed almost 1:1; open prose decays after position 1 and gains marginally.
• Boot with the draft takes ~4-8 min (graph reserve + checkpoint buffers). Perplexity sanity on our port: identical to the pre-MTP build (PPL 1.0952 on the memorized holdout — trunk untouched).
• Vision (--mmproj) works together with the MTP drafter (2026-08-28 fix, same patch lineage): image chunks are not replayed on the MTP draft context — the drafter still sees the image through the trunk hidden state, and the target's verify keeps the output exact. Acceptance on vision requests measured 98.3%.
• RS ring-salvage rollback enabled for qwen4exp (2026-08-28, same patch lineage): partial verify rejections roll back per-row instead of a full ~115-148 MiB checkpoint restore, so the drafter never stalls at long agent contexts (speculative replay stalled gone). An earlier note here described a "one-round acceptance dip after each rollback" as the ring's trade-off: that dip was actually the symptom of the missing conv/PLE ring-slot writer, fixed by patch 0012 (2026-08-30) — with the fix the dip is gone, draft acceptance rises from ~0.74 to 0.91-0.95, and decode gains ~+70% at n=6 (41-44 tok/s on code workloads, Vulkan). The ring itself costs ~7.2 GiB of buffer.

Quality on agent workloads — cleared (root-caused 2026-08-30, fixed)

Earlier agent sessions with this quant showed output-integrity incidents (reasoning

breaking mid-sentence, digit loss inside hex/number runs, malformed tool calls, premature

stops). Root cause: a runtime bug in our fork, not the quant. The qwen4exp

conv-history writer restored the GDN conv state and the PLE history from never-written

ring slots after every speculative rollback — plain decoding never touched that path,

which is why non-speculative builds of the same quant were clean. Fixed by patch 0012

(branch qwen4exp-conv-ring-slots): after the fix, identical agent-style probes produce

zero anomalies, draft acceptance is 0.91-0.95 and decode reaches 41-44 tok/s at n=6 on

Vulkan (see the spec-decoding section). **This quant is recommended for agent deployments

too — on a runtime that includes patch 0012.**

Backend: Vulkan (RADV) beats ROCm/HIP by ~20% (measured, 2026-08-29)

Same machine (Radeon 8060S, Strix Halo), same fork build (commit 2c6309e3b), same protocol (4 prompts × 3 reps, greedy, external MTP drafter Q8_0, --parallel 1, --no-mmap, ctx 8192, p_min 0.75, n_max 3):

| tok/s (median) | ROCm/HIP | Vulkan/RADV | delta |

|---|---|---|---|

| prose (Roma) | 26.0 | 31.8 | +22% |

| prose (Rinascimento) | 25.9 | 31.6 | +22% |

| counting 1-200 | 44.0 | 47.0 | +7% |

| alphabet | 30.4 | 36.7 | +21% |

• Speculative acceptance is identical on both backends (mean accepted length 2.98 vs 2.94): the gap is pure backend speed, not drafter behavior.
• Rep-to-rep variance collapses on Vulkan (about ±10% down to ±0.6% with --no-mmap): with mmap the weights page in lazily and occasionally stall generation; --no-mmap pins them up front. On ROCm --no-mmap measured neutral.
• Recommendation: serve this model on Vulkan/RADV on gfx1151. For context, on the dense Qwen3.8-27B the same test harness measured the two backends at parity — the Vulkan advantage shows on this PLE-hybrid MoE architecture.
• n_max: 6 is the measured optimum. 8 regresses (~35 vs ~42 tok/s on code workloads, re-measured after patch 0012 — the marginal draft positions 7-8 do not pay for themselves); n_max 8 is clean output-wise on fixed builds but slower.

Quant comparison: STRIX_LEAN vs Unsloth UD-IQ4_XS (measured, 2026-08-29)

Same fork build (commit 2c6309e3b, Vulkan/RADV), same protocol as the backend test above (4 prompts × 3 reps, greedy, external MTP drafter Q8_0, n_max 3, p_min 0.75, ctx 8192, --no-mmap, -fit off), same machine. Arms differ only in the target GGUF: this STRIX_LEAN vs unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS (87.2 GiB sharded / 93.7 GB merged):

| tok/s (median) | STRIX_LEAN | UD-IQ4_XS | Δ (UD vs LEAN) |

|---|---|---|---|

| prose (Roma) | 31.5 | 27.8 | -11.7% |

| prose (Rinascimento) | 31.0 | 26.5 | -14.5% |

| counting 1-200 | 45.5 | 44.0 | -3.3% |

| alphabet | 36.0 | 34.7 | -3.6% |

• Speculative acceptance is identical across quants (mean accepted length 3.0-3.25): the gap is pure decode speed of the quantized kernels, not drafter behavior.
• Engine note (updated 2026-08-30): a clean rebuild of master (c841aeeb8) loads the unsloth GGUF fine — sharded or merged, with or without -fa/--no-mmap; the earlier failure was a stale local build, not upstream and not the quant. What does NOT work on official today is the speculative path: with -md <MTP head> --spec-type draft-mtp the target itself fails to load (check_tensor_dims: blk.0.hc_attn_norm.weight not found in ms). So: plain serving of Flash-Next quants works on official, the external-drafter setup requires our fork.
• Verdict: STRIX_LEAN keeps the throughput edge on every workload (prose included — the one that matters for agents) and remains the only path with integrated vision (mmproj), context checkpoints and reasoning-budget on our fork.

Tooling & credits

• Architecture support ported from upstream ggml-org/llama.cpp PR #27742 (danielhanchen) to the ROCmFPX fork (converter + quantizer side); MTP/NextN draft-head support ported from PR #27836 (same author, draft stage).
• Quantization pipeline and calibration by pugant; conversion work assisted by GLM by z.ai.
• Reference quant by kingjones777 (same preset, no imatrix) used as compatibility baseline.