shefowl/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-Hybrid-GGUF overview

---

license: other

license_name: qwen-community-1.0

license_link: LICENSE

base_model:

• SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF

base_model_relation: quantized

pipeline_tag: image-text-to-text

tags:

• gguf
• strata
• llama.cpp
• moe
• quantized
• abliterated
• vulkan

---

<div align="center">

<h1 style="text-align: center;">Qwen3.8-Flash-Next · GSQ-RCO abliterated · Hybrid</h1>

IQ3_XXS trunk, Q2_0 experts: 10-20% faster than IQ3_XXS on Strata, the perplexity of IQ3_XXS on our Vulkan fork

<p><a href="https://github.com/Niko1221/Strata"><img style="display: inline-block; margin: 0 2px;" src="https://img.shields.io/badge/runs%20on-Strata-2563eb" alt="runs on Strata"></a>&nbsp;<a href="https://github.com/shefowl/llama.cpp/tree/vulkan-moe-hot/tools/moe-hot"><img style="display: inline-block; margin: 0 2px;" src="https://img.shields.io/badge/Vulkan-vulkan--moe--hot%20fork-64748b" alt="Vulkan: vulkan-moe-hot fork"></a>&nbsp;<img style="display: inline-block; margin: 0 2px;" src="https://img.shields.io/badge/size-67%20GB-64748b" alt="67 GB">&nbsp;<img style="display: inline-block; margin: 0 2px;" src="https://img.shields.io/badge/license-Qwen%20Community%201.0-64748b" alt="Qwen Community License 1.0"></p>

</div>

<div style="border-left: 3px solid #2563eb; padding: 6px 14px; margin: 18px 0;">

<b>Run it with <a href="https://github.com/Niko1221/Strata">Strata</a>.</b> On our RX 7900 XTX, Strata ran this file at 53-89 tokens/s, 1.6-2.3x the speed of our own fork, and read prompts about 6x faster (measured with IQ3_XXS). Our llama.cpp fork <a href="https://github.com/shefowl/llama.cpp/tree/vulkan-moe-hot/tools/moe-hot">vulkan-moe-hot</a> stays as the option for Vulkan (no ROCm or CUDA). With the fork, the hot experts come from the IQ3_XXS file, which gives this file the perplexity of IQ3_XXS.

</div>

<div style="border-left: 3px solid #64748b; padding: 6px 14px; margin: 18px 0;">

<b>Small machine?</b> This file ran at 9.0 tokens/s on a laptop with an RTX 4050 (6 GB) and 16 GB of RAM; the experts that do not fit in memory are read from the SSD. A copy with <a href="https://huggingface.co/shefowl/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-Hybrid-352E-4050-16GB-Generalist-GGUF">352 of the 512 experts</a> ran there at 10.8 and gives up about a seventh of the factual knowledge. The tables are under <a href="#small-machines-6-to-16-gb-of-vram-with-16-gb-of-ram">Small machines</a>.

</div>

What it is

One GGUF assembled from two tiers of SC117's abliterated GSQ-RCO quants of Qwen3.8-Flash-Next:

• every non-expert tensor (attention, shared experts, embeddings, norms) is from the IQ3_XXS tier;
• every routed expert tensor (ffn_{gate,up,down}_exps) is from the Q2_0 tier.

Nothing was re-quantized: the tensors are copied byte for byte (gguf_swap_experts.py). The Q2_0 tier has the smallest experts (1.44 MB per expert against 1.75 MB in IQ3_XXS), but it also cheapens the dense part: shared experts down to Q2_0, attention to Q3_K, ple_key from BF16 to Q2_0. That part is only 0.38 GB larger in IQ3_XXS, and it carries most of the Q2_0 tier's quality loss.

What you get depends on the runtime:

| runtime | routed experts | perplexity against IQ3_XXS |

|---|---|---|

| Strata, stock llama.cpp | all Q2_0 | +2.6% (the Q2_0 tier: +4.3%) |

| our vulkan-moe-hot fork | hot ones IQ3_XXS (read from SC117's IQ3_XXS file), cold ones Q2_0 | -0.2 ± 0.8% |

Running it

With Strata (recommended)

Strata is an engine made for this model: NVIDIA or AMD, Windows or Linux. Its installer sets up its own model files. To run this file, pack it with Strata's tools. We used Strata at commit 99f3dbd on Linux, with the HIP backend for the RX 7900 XTX (docs/AMD_HIP.md has the build).

# in the Strata folder, with the engine built
.venv/bin/python tools/iq_pack.py --gguf /path/to/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-00001-of-00002.gguf \
  --out packs/hybrid --experts-bin
# the MTP draft layer, as in Strata's docs/ORCA.md
.venv/bin/python tools/mtp_fetch.py fetch --out mtp
.venv/bin/python tools/mtp_pack.py --src mtp --experts q2_0 --out mtp/mtp-q2_0.gguf
.venv/bin/python tools/mtp_rt.py --gguf mtp/mtp-q2_0.gguf --out mtp/rt
cp data/draft_vocab_cyrillic.bin mtp/rt/draft_vocab.bin   # or data/draft_vocab.bin, the default (CJK)

strata-hybrid.json in the Strata folder:

{
 "exe": "build-hip/strata",
 "args": ["--pack", "packs/hybrid",
  "--native", "/path/to/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-00001-of-00002.gguf",
  "--ple-gguf", "/path/to/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-00002-of-00002.gguf",
  "--expert-profile", "data/expert-profile.bin", "--expert-cache", "auto", "--prefill", "auto",
  "--spec", "4", "--spec-min-p", "0.5", "--mtp", "mtp/rt",
  "--max-context", "32768", "--kv", "int8", "--resident-experts", "--vram-reserve-mib", "3072"],
 "cwd": ".", "tokenizer": "packs/hybrid/tokenizer", "model_name": "qwen3.8-flash-next-hybrid",
 "log": "strata-hybrid.log", "host": "127.0.0.1", "port": 8080
}

.venv/bin/python -m serve.server --engine strata --config strata-hybrid.json --port 8080

• --vram-reserve-mib 3072 is for a card that also drives a desktop: with 1900, only about 650 MiB of VRAM stayed free. Strata's setup suggests 3072 for such cards.
• --resident-experts locks the experts that are not in VRAM in RAM: 19.9 GiB for this file, 28.4 GiB for IQ3_XXS. Raise the memlock limit for that.
• The draft vocabulary is a subset of tokens for the MTP layer. The Cyrillic one holds English, code and Cyrillic; the default holds Chinese, Japanese and Korean.
• Pictures on AMD. Strata's setup builds the image encoder (tools/vision) for the CPU there. In our runs that took about 4 s per picture at ~300 image tokens. Built with -DGGML_HIP=ON -DGPU_TARGETS=gfx1100, the encoder ran on the GPU: about 0.5 s per picture at 1,024 image tokens. It took ~1.3 GiB of VRAM from the expert cache, so code generation was 9% slower and other text about the same. The mmproj of the original GGUFs works.

With the vulkan-moe-hot fork (Vulkan)

shefowl/llama.cpp, branch vulkan-moe-hot keeps copies of the most used ("hot") experts of every layer in VRAM and computes the rest ("cold") on the CPU from RAM. Made and tested on an RX 7900 XTX with RADV. With LLAMA_MOE_HOT_SRC the hot copies are read from another GGUF of the same model. This gives per-expert precision, which GGUF itself cannot express:

| | precision | read from |

|---|---|---|

| dense weights | IQ3_XXS | this file |

| hot experts, in VRAM | IQ3_XXS | SC117's IQ3_XXS shard 1, which you need too (47 GB) |

| cold experts, on the CPU | Q2_0 | this file |

LLAMA_MOE_HOT_SRC=Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-00001-of-00002.gguf \
MODEL=Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-00001-of-00002.gguf \
HOT_LIST=hot-lists/hot-12.5GB-no-vision.txt \
MTP=mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf \
tools/moe-hot/run-hot.sh

run-hot.sh is in the fork. The MTP head is from drluoto/Qwen3.8-Flash-Next-MTP-GGUF.

The lists in hot-lists/ are for a 24 GB card that also drives a desktop (about 3 GB): 12.5 GB of hot experts without vision, 11 GB with the mmproj. For another budget, make one with tools/moe-hot/moe_hot_list.py and give it the IQ3_XXS file as --model, because the hot copies come from there. The fork's README has the details.

With stock llama.cpp

Load shard 1 as usual; every routed expert is then Q2_0. Upstream llama.cpp has no SIMD code for Q2_0 on x86, so the dot product falls back to scalar code (48 cycles per 32 weights on Zen 4), and experts kept on the CPU (-ot exps=CPU) run slowly. The fork has AVX2 and AVX-512 VBMI versions (5.0 and 3.4 cycles).

Measurements

One machine: RX 7900 XTX 24 GB, Ryzen 7 7700X, 64 GB DDR5, GPU power level high. VRAM left free: about 1 GB under Strata, about 1.9 GB under the fork.

Strata against our fork

Decode t/s: five 2000-token greedy answers, one per topic, one after another in one session, with MTP. Strata used the Cyrillic draft vocabulary. The fork used hot-set adaptation, its best setting for long answers.

| t/s | code | science | English prose | Cyrillic (Russian) | Chinese |

|---|---|---|---|---|---|

| this hybrid on Strata | 88.8 | 79.7 | 70.8 | 75.1 | 53.3 |

| SC117's IQ3_XXS on Strata | 80.7 | 66.5 | 64.5 | 68.4 | 48.6 |

| this hybrid on vulkan-moe-hot | 54.1 | 43.8 | 30.6 | 34.9 | 33.9 |

• Prompt reading, fresh 5k- and 8k-token prompts: Strata 1,500 and 1,460 tokens/s (with IQ3_XXS), the fork 263 and 244 tokens/s (with this hybrid).
• Experts in VRAM under Strata: 11,871 of this file's or 9,431 of IQ3_XXS's, 15.3 GiB either way.
• The default draft vocabulary (CJK) gave IQ3_XXS 62.1 t/s on Chinese and 49.6 on Cyrillic.
• Repeatability. One Strata session per row. Repeated Strata runs of IQ3_XXS varied by up to 8%, and Strata's output does not repeat bit for bit from run to run.

Small machines: 6 to 16 GB of VRAM with 16 GB of RAM

When the experts do not fit in VRAM plus RAM, Strata reads the rest from the SSD for every token, and speed is roughly the SSD's read rate divided by the megabytes read per token. The rows below compare this file with its pruned copy, 352 of 512 experts per layer. Five 2000-token greedy answers on a freshly started engine with Strata's own expert order.

A real laptop, 6 GB + 16 GB. Acer Nitro V 16 (ANV16-41): Ryzen 5 8645HS, RTX 4050 Laptop 6 GB, 2x8 GB DDR5-5600, Kingston OM8PGP4512Q OEM NVMe (1.4-1.6 GiB/s in the engine's read pattern). Nobara Linux, Strata 0.1.42 built for CUDA, 8K context, no MTP draft layer.

| t/s | code | science | English prose | Cyrillic (Russian) | Chinese | mean |

|---|---|---|---|---|---|---|

| this file, 512 experts | 8.0 | 9.8 | 9.6 | 9.4 | 8.3 | 9.0 |

| 352 experts | 9.4 | 11.7 | 11.5 | 11.9 | 9.7 | 10.8 |

Six more code tasks, 1200 tokens each:

| t/s | first start | after the engine learned the workload |

|---|---|---|

| this file, 512 experts | 7.3 | 10.0 |

| 352 experts | 9.3 | 13.3 |

"Learned the workload": Strata saved which experts one code session used (--expert-profile-save) and loaded them first the next time. The same profile cost 16-26% on other text.

Larger cards, emulated. Our desktop held to the budget: Strata limited to the given VRAM, engine and server in a cgroup with 12 GiB of RAM and no swap, 6 cores, 32K context, MTP draft layer on.

| mean t/s | this file, 512 experts | 352 experts |

|---|---|---|

| 8 GB + 16 GB | 11.1 | 14.7 |

| 12 GB + 16 GB | 30.4 | 43.2 |

| 16 GB + 16 GB | 48.6 | 62.7 |

The six code tasks on 8 GB + 16 GB: this file 8.3 t/s on the first start and 17.4 after the engine learned the workload; the 352-expert copy 12.2 and 24.4.

• A long context costs VRAM. With --max-context 262144 the K/V cache takes about 4 GiB from the expert cache: 12 GB + 16 GB fell from 30 to 14 t/s and 16 GB + 16 GB from 49 to 33, with short prompts.
• What the pruned copy gives up, same tests on the same machine:

| | facts: NQ-open, 1800 questions | code: HumanEval + MBPP, 591 tasks | loops in 100 code answers, greedy / sampled |

|---|---|---|---|

| this file, 512 experts | 33.1% | 83.6% | 6 / 4 |

| 352 experts | 28.4% | 80.4% | 28 / 12 |

• Settings. The engine settings for 16 GB of RAM are in the 352-expert card and apply to this file too: build the pack without --experts-bin (the engine then reads the experts from the GGUF in place), keep Strata's shipped data/expert-profile.bin, and run with --mmap-experts --resident-budget-gib 11 in place of --resident-experts.

vulkan-moe-hot: the hybrid against other quants

Decode t/s: greedy, 400 tokens, one prompt per language or topic, second pass. Cold experts locked in RAM, MTP with 3 draft tokens, 32k context, hot set fixed (no adaptation).

| t/s | hot experts | code | science | English prose | Cyrillic (Russian) | Chinese |

|---|---|---|---|---|---|---|

| this hybrid | 12.6 GB, 71.6% of calls | 54.5 | 40.1 | 25.5 | 27.2 | 23.1 |

| GSQ-RCO IQ3_XXS | 12.0 GB, 69.9% | 48.6 | 34.4 | 19.6 | 23.3 | 18.4 |

| GSQ-RCO Q2_0 tier | 12.9 GB, 80.1% | 65.8 | 47.5 | 29.8 | 30.5 | 27.0 |

| AD-4.27 (AtomicChat recipe, Navin-Models uncensored) | 11.5 GB, 63.4% | 43.8 | 28.6 | 17.7 | 20.2 | 17.1 |

On the fork, the hybrid is 12-30% faster than IQ3_XXS and 24-44% faster than AD-4.27.

• More hot experts. At the same free VRAM, the hybrid holds 0.6 GB more hot experts than IQ3_XXS: the prefill buffer holds the cold experts of one layer, and Q2_0 ones are smaller.
• RAM. Cold experts locked in RAM: 22.4 GiB for the hybrid, 28.9 for IQ3_XXS, 19.7 for Q2_0, 36.5 for AD-4.27.
• Noise. Run to run the numbers move by about 5%, between sessions by up to 10%: how much VRAM the desktop takes changes the hot list.

With hot-set adaptation (LLAMA_MOE_HOT_ADAPT=256, as run-hot.sh sets it), the fork gets faster as an answer goes on. Over 2000-token answers, t/s after the first 1000 tokens, mean of two sessions:

| | code | science | English prose | Cyrillic (Russian) | Chinese |

|---|---|---|---|---|---|

| adaptation off | 53.2 | 39.6 | 29.1 | 29.7 | 25.0 |

| adaptation on | 56.9 | 48.9 | 33.7 | 39.3 | 36.1 |

Quality against GSQ-RCO IQ3_XXS

Same model, so the differences come from the quantization alone. Perplexity on 20k tokens of mixed English documentation, C++ and Russian text (40 chunks of 512), paired with the IQ3_XXS logits; ± is the standard error. Measured with llama.cpp.

| | perplexity ratio | mean KLD | same top-1 token |

|---|---|---|---|

| this hybrid on vulkan-moe-hot | 0.998 ± 0.008 | 0.219 | 84.0% |

| this file with every expert Q2_0 (as Strata and stock llama.cpp run it) | 1.026 ± 0.011 | 0.363 | 79.4% |

| GSQ-RCO Q2_0 tier | 1.043 ± 0.011 | 0.405 | 78.1% |

The hot/cold split by itself changes nothing: IQ3_XXS with another hot list gives a KLD of 0.000. The KLD of the hybrid on the fork comes from its cold Q2_0 experts, which move the distribution without moving the perplexity.

Tasks, reasoning_effort medium, greedy, the hybrid on the fork against IQ3_XXS:

| | hybrid | IQ3_XXS | |

|---|---|---|---|

| GSM-Plus, 100 tasks | 78 | 78 | the same answer on every task |

| CRUXEval-O, 100 tasks | 94 | 97 | 5 character-level slips by the hybrid (a count off by two, a dropped character, a letter case); not significant at n = 100 |

Vision works with the mmproj of the original GGUFs, on both runtimes: 14 px text in a 1280x960 picture was read exactly.

Not measured: quality on Strata (same weights, but Strata's arithmetic is its own), standard perplexity sets (wikitext), other hardware, refusal behaviour.

Files

| file | size | content |

|---|---|---|

| Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-00001-of-00002.gguf | 38.4 GB | IQ3_XXS trunk, Q2_0 routed experts |

| Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-00002-of-00002.gguf | 28.8 GB | per-layer n-gram embedding table, byte-identical to shard 2 of SC117's tiers |

| hot-lists/ | | expert lists for the vulkan-moe-hot fork |

Credits and license

• Qwen: the Qwen3.8-Flash-Next base model, under the Qwen Community License 1.0 (see LICENSE). The upstream GSQ-RCO repositories are tagged Apache-2.0; this repository follows the base model's license.
• IST-DASLab: GSQ and RCO, and the quantized weights.
• orcarouter: the abliterated weights. SC117: the abliterated GSQ-RCO GGUFs this file is assembled from.
• Niko1221 and contributors: Strata (MIT).
• drluoto: the MTP head GGUF for the fork.
• llama.cpp: the GGUF format and the runtime.

Not affiliated with any of them. The model is abliterated: its refusals were removed upstream, and you are responsible for how you use it.