---

base_model: turboderp/Qwen3.8-Flash-Next-exl3

base_model_relation: quantized

quantized_by: turboderp

library_name: gguf

license: other

license_name: qwen-community-1.0

license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE

pipeline_tag: text-generation

tags:

• exl3
• ranma

---

Qwen3.8-Flash-Next RANMA EXL3 GGUF

> [!WARNING]

> Requires ranma.cpp ranma_20261005 or newer (github.com/saturnsky/ranma.cpp).

> These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

What this is

This repository repackages the EXL3 quantizations made by turboderp of

Qwen/Qwen3.8-Flash-Next into GGUF files that ranma.cpp can read.

The weights were not re-quantized. The EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container

(with the fork's EXL3 tensor types) and split into shards. All credit for the quantization goes to turboderp and the

ExLlamaV3 project. Thank you.

Sources (EXL3, by turboderp), one branch per quantization. Each source branch maps to one folder in this repository:

| Source branch (turboderp) | Folder in this repo |

|---|---|

| 3.05bpw_h5_ng5 | EXL3-3.05bpw-h5-ng5 |

| 4.05bpw_h6_ng6 | EXL3-4.05bpw-h6-ng6 |

Both were produced with ExLlamaV3 quantization version 1.4.4 (codebook mul1, calibration 250 rows x 2048). h5/h6 are the head bits

(output layer), ng5/ng6 the n-gram embedding bits.

Not included

These files hold the text model only. Two parts of the source checkpoints are not carried over:

| Part | Size in the source (3.05 / 4.05) | Status |

|---|---|---|

| Multi-token-prediction (MTP) layers | 1.0 / 1.4 GB | Not converted: the EXL3 checkpoint stores the MTP input projection as two separate EXL3 groups, which cannot be written as one GGUF tensor yet. MTP runs as a separate draft model (-md), see Running. |

| Vision encoder | 0.5 / 0.5 GB | Not converted: the converter does not handle quantized vision towers yet. Image and video input is not available with these files; no mmproj file is provided. |

In llama.cpp both parts are separate files (an MTP draft GGUF and an mmproj GGUF), so they can be added later without changing the files here.

Files

Each quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors

(llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma; the rest are found automatically.

| Folder | bpw (head / n-gram) | Size | Files |

|---|---|---:|---:|

| EXL3-3.05bpw-h5-ng5 | 3.05 (5 / 5) | 83.5 GB (77.8 GiB) | 3 |

| EXL3-4.05bpw-h6-ng6 | 4.05 (6 / 6) | 105.5 GB (98.2 GiB) | 4 |

File names: Qwen3.8-Flash-Next-EXL3-<bpw>-<head>-<ngram>-0000K-of-0000N.gguf. manifest.json lists the size and SHA-256 of every file.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the

experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the

GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the examples below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only;

there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement).

The first shard is enough for -m; the other shards are found by name.

| Option | Default | Meaning | How to choose |

|---|---|---|---|

| --expert-l1-mib N | 0 (cache off) | VRAM budget in MiB of the expert cache, L1. It also decides which experts go to host memory. | VRAM - (dense weights + KV cache + compute buffers) - margin; see below. |

| --expert-l2-mib N | -1 (unlimited) | Host-RAM budget in MiB of the expert cache, L2. -1: every expert that is not in L1 lives in RAM. A finite value: experts that fit in neither L1 nor L2 stay in the GGUF file on the SSD and are read on demand (Windows + HIP). | Keep -1 if the experts outside L1 fit in the RAM you can spare. Otherwise use the RAM you can spare for the experts, minus what your other programs need; the SSD serves the rest, at some cost in speed. |

| --expert-cache-mode inclusive\|exclusive | inclusive | exclusive keeps one copy of each expert (in VRAM or in RAM); inclusive keeps a RAM copy of every VRAM resident as well. | exclusive when RAM is tight; the benchmarks use it. Windows. |

| --expert-profile-dir DIR | none | Directory where the server stores which experts each request used. Without it the placement stays a fixed random one. | Any writable directory. The placement adapts at request boundaries; about ten requests rebuild a profile. |

| --expert-prefill-swap | off | Use a separate plan while a prompt is processed. | Optional. |

| --spec-type draft-mtp, -md FILE, -ngld 999 | none | Speculative decoding with a multi-token-prediction draft head (-md), run on the GPU (-ngld 999). The MTP layers are not part of these EXL3 files; the benchmarks used mtp-Qwen3.8-Flash-Next-Q8_0.gguf (ggml-org). | Optional, for faster decoding. |

| --expert-cache-draft on\|off | on | The draft head shares the expert cache, with one L1 and one L2 budget for both models. | Keep on. |

| --spec-smart (default on), --spec-smart-store PATH, --spec-draft-n-max N | on, none, 7 | Chooses the draft length at every step from measured times. The store file keeps the estimates across restarts. | Give a store path so a restart is not cold; leave the rest. |

| -np N | 4 in the server | Number of parallel slots. Always give -np 1 for these examples: with the default 4 slots the recurrent state and compute buffers at 256K grow by several GiB, and a finite L2 (--expert-l2-mib positive) is refused when the slot count is not fixed (finite L2 needs positive batch/parallel bounds). | -np 1 unless you serve several users. |

| --gpu-heartbeat-seconds S | 5.0 (server) | Pulses the GPU while the server is idle so that the OS does not evict VRAM. | Leave the default. |

Sizing L1 and L2

• L1. L1 = VRAM - (memory the cache does not cover) - margin. What the cache does not cover is the dense weights, the KV cache and the compute buffers;

the KV cache is proportional to the context. Reference values from the benchmark runs (context 65536, -b 512 -ub 512, flash attention, f16 KV cache): total device memory was 28.4 to 29.5 GiB with L1 = 20 GiB on a 32 GiB card, i.e. about 8.4 to 9.5 GiB outside the cache, and 11.3 to 12.4 GiB with L1 = 3 GiB when emulating a 16 GiB card (again about 8.3 to 9.4 GiB outside).

Keep a margin of a few GiB, more if the GPU also drives a display. The benchmark budgets were deliberately conservative; a larger L1 puts more experts in

VRAM. The L1 budget includes the cache's own tables and overhead, so it is not exactly the expert bytes.

• L2. With -1, the RAM used by the experts is about the model size minus L1 (exclusive mode); the KV cache, prompt cache and buffers come on top.

With a finite L2 the cache's host part is bounded by it. Set it to the RAM you can spare minus what your other programs use. The server refuses to load

if the host requirement is above the physical memory installed, but it does not look at how much is free at that moment.

• If L1 + L2 covers all experts, nothing is read from the file; each expert that has to come from the SSD costs speed.

The commands and values below are examples only. Pick them for your own use and system: the context length (a long roleplay needs a large -c, a short chat or a coding session can use much less and leave VRAM to L1), the split between L1 and L2, whether to use MTP, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and workload works best).

Example commands (256K context)

The KV cache grows in proportion to the context. For this model it is about 30 KiB per token (f16), i.e. about 7.5 GiB at -c 262144 against about 1.9 GiB at 65536, so the L1 values below are lower than the benchmark ones. Checked on an R9700 (32 GB) with -np 1 and MTP: load and a short generation worked; dedicated VRAM peaked at 29.1 GiB (3.05 bpw) and 29.5 GiB (4.05 bpw) with L1 = 13312 (an L1 of 15360 exceeded 31 GiB and was too much). The 64 GB RAM command (finite L2 40960) was checked the same way (29.1 / 29.5 GiB). If the load fails or VRAM runs out, lower L1 by 1-2 GiB.

32 GB VRAM + 128 GB RAM (all other experts in RAM):

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.05bpw-h5-ng5\Qwen3.8-Flash-Next-EXL3-3.05bpw-h5-ng5-00001-of-00003.gguf ^
  -c 262144 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 13312 --expert-l2-mib -1 ^
  --expert-profile-dir .\expert-profiles ^
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf -ngld 999 --spec-type draft-mtp --spec-smart-store .\smart-store.json

32 GB VRAM + 64 GB RAM (finite L2; experts that fit in neither tier are read from the SSD):

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.05bpw-h5-ng5\Qwen3.8-Flash-Next-EXL3-3.05bpw-h5-ng5-00001-of-00003.gguf ^
  -c 262144 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 13312 --expert-l2-mib 40960 ^
  --expert-profile-dir .\expert-profiles ^
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf -ngld 999 --spec-type draft-mtp --spec-smart-store .\smart-store.json

The MTP lines are optional; drop them to run without speculative decoding. A 16 GB card cannot hold a 256K KV cache next to the dense weights: use -c 131072 --expert-l1-mib 2048 -np 1 there (checked on an R9700 with the 16 GB budget emulated: dedicated VRAM peak 12.9 GiB) (same --expert-l2-mib 40960 on 64 GB RAM), or a shorter context.

Benchmark settings (for the tables below)

The benchmark tables use -c 65536 -np 1 with --expert-l1-mib 20480 (32 GB VRAM), --expert-l2-mib -1 (128 GB RAM) or 40960 (64 GB RAM), and --expert-l1-mib 3072 --expert-l2-mib 40960 for the 16 GB card emulation. They are not the example commands above.

Details of every option: expert-cache.md, expert-cache-exclusive.md,

expert-cache-l2.md, host-direct-moe.md, spec-smart.md,

gpu-heartbeat.md.

Benchmarks

Measured on a Radeon AI PRO R9700 (PCIe 5.0 x16, headless) with ranma_20261005.

The RX 9070 XT rows are emulated on the R9700 by limiting the expert cache to the budget of a 16 GiB card. Each row was measured once.

llama-bench TG128 at depth 0 (tokens/s, warm expert cache, exclusive mode):

| System | EXL3 3.05 bpw | EXL3 4.05 bpw |

|---|---:|---:|

| R9700 + 128 GB RAM | 53.5 | 49.6 |

| R9700 + 64 GB RAM | 52.4 | 47.7 |

| RX 9070 XT (emulated) + 64 GB RAM | 43.2 | 37.0 |

MTP smart decoding, server decode speed (tokens/s, range over coding and four roleplay presets; draft head mtp-Qwen3.8-Flash-Next-Q8_0.gguf):

| System | EXL3 3.05 bpw | EXL3 4.05 bpw |

|---|---:|---:|

| R9700 + 128 GB RAM | 60.8 - 66.3 | 53.9 - 57.0 |

| R9700 + 64 GB RAM | 58.8 - 66.4 | 53.3 - 57.3 |

| RX 9070 XT (emulated) + 64 GB RAM | 39.6 - 42.2 | 29.1 - 30.8 |

Conditions: context 65536, greedy sampling, at most 384 generated tokens for server rows, every process started after a 300 s rest.

Full tables (prompt processing, deeper contexts, cold runs, memory) and the exact method:

ranma.cpp benchmark document.

License

Same as the original model: Qwen Community License 1.0 (qwen-community-1.0), see the

original repository. Use of these files is subject to that license.