---

license: other

license_name: qwen

base_model: Qwen/Qwen3.8-Flash-Next

base_model_relation: quantized

tags:

- gguf

- qwen4_exp

- mxfp4

- llama.cpp

- ik_llama.cpp

- speculative-decoding

- mtp

---

!banner

Qwen3.8-Flash-Next · MXFP4 GGUF · with the MTP head

Quantizations of Qwen/Qwen3.8-Flash-Next

(125B-A6B MoE + 51B-entry n-gram table + 2.6B MTP head — the qwen4_exp

architecture preview of Qwen4), built directly from the official BF16 release

and tuned for single-GPU + CPU hybrid serving: a 24-32 GB GPU carrying

attention and a slice of the experts, system RAM carrying the rest.

What makes these files different from other quants of this model:

1. The MTP head is included. The checkpoint ships a 2.6B multi-token-

prediction head that every public converter currently drops. The -MTP

files here keep it (33 tensors, appended with converter-faithful

transforms), enabling lossless speculative decoding — measured

0.93–0.99 draft acceptance on real traffic.

1. The n-gram (PLE) table is treated with respect. It's a row-lookup

table with random access patterns — aggressive quantization damages it

disproportionately. Here it's either BF16 (full quality, mmap-friendly)

or Q8_0 (measured +0.12% perplexity, and small enough that a

128 GB-RAM machine holds the whole CPU side in memory).

1. MXFP4 routed experts — fast on CPU (AVX-512) and GPU alike, with

attention/router/shared-expert kept at Q8_0/F32.

Which file do I want?

| file | size | choose it when |

|---|---|---|

| ...-ngramBF16.gguf | 160.2 GiB | You have >128 GB RAM or rely on mmap, and run stock llama.cpp (merged) or ik_llama.cpp (merged). Maximum table fidelity. |

| ...-ngramBF16-MTP.gguf | 162.8 GiB | Same as above plus MTP speculative decoding (needs the MTP engine branch, below). |

| ...-ngramQ8-MTP.gguf | 118.1 GiB | Recommended for 128 GB-RAM machines. The whole CPU side fits in one pinned allocation — no paging, tight run-to-run variance — with MTP included. (+0.12% PPL vs BF16 table.) |

Quick start (stock llama.cpp or ik_llama.cpp, no MTP)

llama-server -m Qwen3.8-Flash-Next-MXFP4-ngramBF16.gguf \
  -ngl 999 -ncmoe 38 -fa 1 -c 65536 -ctk q8_0 -ctv q8_0 \
  -t <physical cores> --jinja

• -ncmoe N keeps N layers' routed experts in system RAM; tune to your

VRAM (38 fits a 32 GB card with ~10 GB headroom at 64K context; the full

262K window also allocates).

• Sampler defaults (thinking mode): temp 1.0, top-p 0.95, top-k 20.
• The template defaults to reasoning_effort: xhigh. For faster answers:

--chat-template-kwargs '{"reasoning_effort":"low"}' (llama.cpp), or

--reasoning-budget N (ik_llama).

Speculative decoding (the fun part — ik_llama + MTP file)

llama-server -m Qwen3.8-Flash-Next-MXFP4-ngramQ8-MTP.gguf \
  -ngl 999 -ncmoe 38 -fa 1 -c 65536 -ctk q8_0 -ctv q8_0 \
  -t 24 -tb 32 -rtr -muge --jinja \
  --spec-type ngram-mod:n_min=4 --spec-type mtp:n_max=4

Two draft stages chain: ngram-mod supplies long drafts on repetitive

spans (config edits, code echoes), the MTP head drafts everything else at

0.93–0.99 acceptance. Both are lossless — the target model verifies

every drafted token. -rtr (runtime repack for AVX-512) and -muge

(merged expert tensors) are each independently measured wins on the CPU

expert path.

The MTP stage currently requires the qwen4exp-MTP engine branch

(upstream submission to ik_llama.cpp in progress); without it the -MTP

files still serve normally on merged ik_llama main — the extra tensors

are simply skipped unless MTP is requested.

Measured performance

RTX 5090 (32 GB) + Threadripper 9960X (24C, quad-channel DDR5-5600),

Q8-table MTP file, config above:

| workload | throughput |

|---|---|

| edit-heavy agentic decode (spec on) | ~66 tok/s (peaks 70+) |

| general decode (spec on) | ~57 tok/s |

| plain decode (no speculation) | ~45 tok/s |

| prefill | ~700 tok/s at -ub 512 (higher with larger -ub) |

Rule of thumb for other machines: decode scales with RAM bandwidth for

the CPU-resident experts (only ~6 GB of active weights stream per token)

plus whatever your GPU carries; a 24 GB card with 96 GB RAM runs the

Q8-table file with -ncmoe raised accordingly.

Recipe notes

• Quantized with the llama.cpp qwen4exp toolchain from official BF16;

tensor-type rules keep router/norms F32, attention/shared-expert Q8_0,

routed experts MXFP4, n-gram table BF16 or Q8_0.

• The MTP head: 33 tensors appended by GGUF surgery with the converter's

exact conventions (zero-centred norm +1 baking, fused-tensor splits),

block_count=49, nextn_predict_layers=1, byte-fidelity audited

against the source shards. MTP-layer experts kept at Q8_0 (draft

accuracy is what makes speculation pay).

• Verified: perplexity within noise of the reference conversion, and the

Q8-table delta measured at +0.12% on a fixed corpus.

*Qwen model license applies (see base model). Quantization + MTP

packaging by @jamesrogers.*