---

license: other

license_name: qwen-community-license-1.0

base_model: Qwen/Qwen3.8-Flash-Next-FP8

library: gguf

quantized_by: julianmb

pipeline_tag: text-generation

tags:

• qwen4exp
• strix-halo
• rocmfpx
• gguf
• ple-quantized

---

Qwen3.8-Flash-Next GGUFs — provenance-verified quants for Strix Halo

Four files:

| file | size | what it is |

|---|---|---|

| Qwen3.8-Flash-Next-IQ4_XS-PLE.gguf | 91 GiB | recommended daily driver — iq4_xs trunk with the 27G PLE n-gram table at iq4_nl |

| Qwen3.8-Flash-Next-IQ4_XS.gguf | 116 GiB | static reference quant, PLE at q8_0 |

| Qwen3.8-Flash-Next-IQ4_XS-M2.gguf | 115 GiB | imatrix-calibrated quant, PLE table at q8_0 — best measured perplexity |

| mtp-Qwen3.8-Flash-Next-Q8_0.gguf | 3.9 GiB | MTP draft sidecar for nathanw1014-lineage engines (fork-specific — will NOT load on apepojken/mainline) |

M2 — the imatrix quant

second-generation quant: same trunk type (iq4_xs), but calibrated with a

926-entry imatrix (1,024 chunks) via the ROCmFPX banded quantizer, and the

51B PLE lookup table left at q8_0 (no --tensor-type cut). 5.56 bpw,

115 giB — 24 giB bigger than the 91g PLE file.

perplexity (wiki.test.raw, ctx 2048, 145 chunks):

| quant | PPL |

|---|---|

| M2 (imatrix, PLE q8_0) | 4.2809 ±0.025 |

| PLE 91g | 4.2932 ±0.025 (statistically tied, <0.5σ) |

| static 116g | 4.5221 ±0.026 (~9σ worse) |

speed profile is honest-mixed (single runs, nathanw1014 vulkan engine,

q8_0 kv, temp 0):

| depth | plain tg | mtp tg |

|---|---|---|

| 8k | 23.8 (≈ PLE 23.8) | 25.1 (PLE 33.5 — M2 slower) |

| 32k | 19.0 (≈ PLE 19.0) | 29.1 (best of the three) |

| 128k | — | 13.3 (PLE 13.5 — tied) |

pick M2 when you want the best measured quality and don't mind the

24 giB: at ≤32k plain it matches PLE, and at 32k MTP it measured fastest.

for shallow-depth MTP speed take the 91g PLE; for deep 128k+ MTP the

static 116g had a small in-sweep edge (17.4 vs 13.3/13.5 — within the

same-config spread, n=1 caveat).

provenance

every quant descends from an F16 that was byte-verified against the official

Qwen/Qwen3.8-Flash-Next-FP8 checkpoint: hyper-connection norms folded to

(1 + w) (97/97 tensors — the converter bug that produces deterministic garbage

is fixed in our pipeline), PLE fp8 scale applied, expert stacking identity

probed 512x3, GDN v-head reorder checked. details:

https://github.com/julianmb/haloq38flash

the PLE cut (what makes the 91G special)

the 51B-parameter n-gram lookup table was moved from q8_0 (54G) to iq4_nl

(27G) via --tensor-type. hash-gathered lookup rows tolerate the precision

drop — verified by smoke and full benchmark, no degradation observed:

| depth | static 116G plain/mtp t/s | PLE 91G plain/mtp t/s |

|---|---|---|

| 0 | 29.2 / 48.4 | 29.9 / 53.1 |

| 8k | 22.9 / 42.8 | 24.1 / 56.4 |

| 32k | 19.5 / 29.5 | 20.1 / 30.2 |

prefill at 32k: 384 → 397 t/s. no collapse at depth. engine: nathanw1014

strix-halo-vulkan (ad914eb), vulkan/radv, q8_0 KV, -ub 2048, temp 0.

fork compatibility caveat (important)

the iq4_nl PLE rows assert in SOME forks: engines that feed gathered PLE

rows directly as mul_mat B operands without dequantizing (apepojken

qwen4exp-spec-mtp) abort at ggml-vulkan.cpp:7794 (b_type must be

F32/F16/Q8_1). verified working on nathanw1014 strix-halo-vulkan. if your

engine asserts on load or first token, use the 116G static file instead.

M2 keeps the PLE table at q8_0 and has no such assert exposure.

provenance note

the same --tensor-type cut applied to unverified-source quants will NOT fix

a broken converter (hc norms, PLE scale) — garbage in, garbage out. ours is

built from a fixed, audited pipeline.

128k & 256k context benchmarks (SSD-PLE)

the depth story is not monotonic. measured on the same engine (nathanw1014

vulkan, q8_0 kv, temp 0):

| depth | static 116g plain/mtp t/s | PLE 91g plain/mtp t/s |

|---|---|---|

| 0 | 29.2 / 48.4 | 29.1 / 48.1 |

| 8k | 22.9 / 42.8 | 21.8 / 27.9 (peak: 56.4) |

| 32k | 19.5 / 29.5 | 18.5 / 25.5 (peak: 30.2) |

| 128k | 10.8 / 26.9 | 8.9 / 11.8 (peak: 18.6) |

| 256k | 6.2 / — | 6.0 / 15.2 (SSD-PLE tuned) |

• 256k context unlocked with SSD-PLE: With -lm mmap --tensor-read-lazy on, the 27 GB PLE table is kept on NVMe SSD, keeping active RAM under 95 GB and providing ~30 GB of free headroom at full 256k context. Combined with -tb 16 and -ub 1024 -b 2048 -t 4, this achieves 15.2 t/s MTP generation and 179-191.5 t/s prefill at 256k with zero swap thrashing.
• Hardware boundary: Do not use -ub 2048 at 256k context; it triggers Vulkan queue submission timeouts (ErrorDeviceLost). -ub 1024 is the tested maximum.
• 128k reversal: At 128k specifically under MTP, the static 116g quant has higher draft acceptance than the 91g PLE quant (26.9 vs 18.6 t/s).
• Practical: Use the 91g PLE file for daily chat, coding, and full 256k long context (with SSD-PLE flags). Use the 116g static file if you specifically need highest MTP speed at 128k.

run it (strix halo, 128 GB unified memory)

full methodology, receipts, and the benchmark record:

https://github.com/julianmb/haloq38flash

docker (one-liner; image default serves the 91g PLE on :8080):

git clone https://github.com/julianmb/haloq38flash && cd haloq38flash
docker compose up --build

point it at M2 with the MTP sidecar and the warm-turn cache:

docker compose run qwen38-flash-next /app/llama-server \
  -m /models/Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md /models/mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  --cache-ram 8192 --ctx-checkpoints 32 \
  -c 32768 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4

or raw llama.cpp (nathanw1014 strix-halo-vulkan lineage engines):

llama-server -m Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4 -c 32768

perf notes: -t 16 lifts prefill up to +43% at 128k (decode indifferent);

the --cache-ram/--ctx-checkpoints warm-turn flags make repeated

context nearly free (measured 438 s → 0.68 s at 128k, 994 s → 0.74 s at

256k — 640x/1351x). swap M2's filename for the other quants; same flags.

license

qwen community license 1.0 (distribution permitted with notice; maas

restrictions apply). base model: Qwen/Qwen3.8-Flash-Next.