---

license: apache-2.0

base_model: Qwen/Qwen3.8-27B

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- llama.cpp

- imatrix

- mixed-precision

- qwen3.8

- rocm

- 24gb

---

Qwen3.8-27B mixed-precision GGUF for 24 GB GPUs at 128k context

Three GGUF quantizations of Qwen/Qwen3.8-27B that spend bits where

quantization error hurts most: the 16 full-attention layers and the recurrent (linear-attention) layers get more

precision, the bulky feed-forward up/gate weights get less. Built for running the model at a **128k context entirely

on one 24 GB GPU** (measured on an RX 7900 XTX with llama.cpp/ROCm).

| File | Size | Use |

|---|---|---|

| Qwen3.8-27B-R3-mix.gguf | 17.0 GiB (18.2 GB) | Most accurate (closest to Q8_0 of all files measured, below). 128k needs a 4-bit KV cache and uses ~20.1 GB of VRAM, the whole budget on a 24 GB card under Windows; MTP layer removed |

| Qwen3.8-27B-R2-mix.gguf | 14.3 GiB (15.4 GB) | Most headroom. 128k context with an 8-bit KV cache in ~19.7 GB of VRAM, or ~17.7 GB with a 4-bit cache; ~15% faster generation than R3 |

| Qwen3.8-27B-R1-mix.gguf | 15.8 GiB (16.9 GB) | Earlier recipe, about the size of a Q4_K_M; superseded by R3 |

All are text-only (no vision projector). Unsloth's mmproj-F16.gguf from

unsloth/Qwen3.8-27B-GGUF is built from the same base model and should

work with them, but this was not tested. R1 and R2 keep the multi-token-prediction (MTP) layer, quantized with the same

recipe; R3 drops it (64 blocks, nextn_predict_layers = 0), so it cannot be used for MTP speculative decoding.

Quality: KL divergence against Q8_0

Reference: a Q8_0 quantization of the same BF16 weights. Text: 24,576 held-out tokens (6 × 4,096) of technical text

(C source, decompiler output, CPU-manual prose, technical notes). Measured with llama-perplexity --kl-divergence.

Lower KLD is closer to the reference.

| Model | Size (MiB) | Mean KLD | Same top token | PPL vs Q8_0 | 99.9th pct KLD |

|---|---|---|---|---|---|

| Ollama library qwen3.8:27b (Q4_K_M) | 16,032 | 0.0336 ± 0.0018 | 92.8% | +2.8% | 2.57 |

| Unsloth UD-IQ4_XS | 13,593 | 0.0227 ± 0.0018 | 94.0% | +0.6% | 1.12 |

| R1-mix (this repo) | 16,167 | 0.0148 ± 0.0017 | 94.6% | +0.9% | 0.54 |

| R2-mix (this repo) | 14,677 | 0.0159 ± 0.0005 | 94.3% | +1.3% | 0.62 |

R1-mix has about 2.3× lower mean KLD than the Ollama library's qwen3.8:27b Q4_K_M (similar size), with a much

smaller worst-case tail; R2-mix about 2.1× lower while 1.3 GB smaller. Only that one Q4_K_M was measured; Q4_K_M

files built with an imatrix (as most community uploads are) were not tested and would likely land closer. Unsloth's UD-IQ4_XS has the best perplexity ratio but a higher KLD;

KLD compares the whole output distribution and is the better quantization metric. Caveats: one evaluation text of

modest size, technical domain only (not chat or tool use), single run. R3-mix was measured later on a larger

evaluation text; its results are in the next section, with R1, R2 and UD-IQ4_XS re-measured on that text.

R3-mix: a recipe from per-tensor measurements

R1 and R2 were designed by hand. For R3 each tensor group was measured instead: starting from a base recipe (R2's

types, MTP layer removed, token_embd at Q8_0 since it stays in system RAM), one group at a time was raised one

quantization step, the file rebuilt and its KL divergence against Q8_0 measured, giving the quality gained per GiB

spent. Median KLD gain per GiB, highest first: full-attention attn_output 0.00080, recurrent ssm_out 0.00055,

attn_q 0.00043, recurrent attn_gate 0.00032, attn_qkv 0.00026, ffn_down 0.00023, ffn_up 0.00022,

ffn_gate 0.00018. Raising the output layer to Q6_K made things slightly worse; spending the bits on one band of

layers instead of all layers did not help. R3 takes the best-value upgrades until the file fills a 24 GB card at 128k

context with a 4-bit KV cache.

Evaluation: 65,536 tokens (16 × 4,096) of technical text (decompiler output, C, notes) interleaved with transcripts

of a coding agent working in that material; reference Q8_0 of the BF16 weights; llama-perplexity --kl-divergence.

Not comparable with the table above (different text). The R1-mix and R2-mix rows are the files in this repo.

| Model | Size (MiB) | Mean KLD | Median KLD | Same top token | PPL vs Q8_0 | 99.9th pct KLD |

|---|---|---|---|---|---|---|

| Unsloth UD-IQ4_XS | 13,593 | 0.0273 ± 0.0017 | 0.00161 | 94.7% | +1.0% | 4.22 |

| R2-mix (this repo) | 14,677 | 0.0199 ± 0.0012 | 0.00131 | 95.1% | +0.5% | 2.46 |

| R1-mix (this repo) | 16,167 | 0.0191 ± 0.0015 | 0.00122 | 95.0% | +0.8% | 2.47 |

| R3-mix (this repo) | 17,390 | 0.0142 ± 0.0011 | 0.00074 | 96.1% | +0.1% | 1.91 |

R3-mix is closest to Q8_0 on every measure: about 30% lower mean KLD and 44% lower median KLD than R2-mix, one more

token in a hundred matching Q8_0's top choice, and a smaller worst-case tail. It is also the largest file. Single

run, one evaluation text; mean KLD varies by about ±0.002 between texts of this size, so the median and top-token

columns are the more stable comparison.

Speed and memory (same machine and llama.cpp build as below, 128k context, KV cache q4_0, batch 2048, three-turn

26.5k + 5.7k + 5.2k-token workload): prompt 961 / 805 / 760 tok/s, generation about 34 tok/s (R2-mix: about 39),

20.07 GB dedicated VRAM, no spill to system memory. That is all the VRAM Windows gave one process on this card, so

close other GPU-heavy applications first. An 8-bit KV cache at 128k does not fit with R3.

Long-context recall (three planted lines at 10/50/90% depth in decompiler output): 3/3 at 57,710 and at

115,936 tokens, with the 4-bit KV cache.

Agent tasks. Through Qwen Code, on the eleven tasks described below: K1-K7 and

E1-E3 all correct, K8 stopped at the turn cap after over-researching (10/11 solved, 16.7 min); in five reruns of K8,

four were fully correct. Through Hermes Agent on a four-task subset

(K1, K2, K4, E1, one run each, a full agent setup with persona, skills and project rules), against two hosted models

through a ChatGPT subscription:

| Task | R3-mix (local, RX 7900 XTX) | OpenAI gpt-6-luna | OpenAI gpt-6-sol |

|---|---|---|---|

| K1 (7 facts) | 7/7, 518 s | 1/7, 168 s | 7/7, 285 s |

| K2 (9 facts) | 9/9, 232 s | 9/9, 81 s | 9/9, 239 s |

| K4 (2 facts) | 1/2, 189 s | 2/2, 45 s | 2/2, 67 s |

| E1 (edit, build, tests) | 6/6, 187 s | 6/6, 526 s | 6/6, 177 s |

| Total | 23/24, 18.8 min | 18/24, 13.7 min | 24/24, 12.8 min |

gpt-6-luna never opened K1's file (twice; it searched elsewhere and declined to answer). R3-mix's K4 miss named the

helper that performs the directory creation rather than the function the answer key expects. Same caps for all (30

turns, 600 s per task); local replies were also capped at 8,192 tokens. Four tasks and single runs: this shows a

local 27B model in the same range as hosted models on this kind of work, not a ranking.

R2-mix vs Unsloth UD-IQ4_XS and Ollama's default

The closest same-model alternative is Unsloth's UD-IQ4_XS, which is smaller; Ollama's library qwen3.8:27b

(Q4_K_M) is what most Ollama users get. Neither alternative is worse on everything:

| | R2-mix (this repo) | Unsloth UD-IQ4_XS | Ollama qwen3.8:27b Q4_K_M |

|---|---|---|---|

| File size | 14,677 MiB | 13,593 MiB | 16,032 MiB |

| Generation speed | 39.9 tok/s | 41.1 tok/s | 36.7 tok/s |

| Prompt processing | 1,012 tok/s | 978 tok/s | 954 tok/s |

| Mean KLD vs Q8_0 | 0.0159 ± 0.0005 | 0.0227 ± 0.0018 | 0.0336 ± 0.0018 |

| Same top token as Q8_0 | 94.3% | 94.0% | 92.8% |

| Perplexity vs Q8_0 | +1.3% | +0.6% | +2.8% |

| 99.9th percentile KLD | 0.62 | 1.12 | 2.57 |

In short: R2-mix stays closest to the full-precision model on average and in the worst cases; UD-IQ4_XS is ~1.1 GB

smaller, ~3% faster at generation and has the best perplexity ratio; the Ollama Q4_K_M is the largest and furthest

from Q8_0 here. KLD/PPL from the quality table above. All three were served the same way for the speed rows: the

mean over a complete agentic test session (next section) with llama-server b1343, ROCm, RX 7900 XTX, 128k context,

KV cache q4_0; prompt speed over chunks of 2k tokens or more. The Ollama column measures those weights under

llama-server, not Ollama's own runtime.

Agentic test through Qwen Code

Eleven tasks run headless through Qwen Code 0.25.1 (qwen -o json), one run each, identical settings for every quant

(llama-server as above, --reasoning-effort medium, temperature 0.3, caps of 30 turns / 10 minutes / 80 tool calls).

K1-K8 are code-reading tasks on a reverse-engineering workspace (Ghidra decompiles, Wine sources, notes), answered as

KEY=value facts and graded against answers checked by hand; E1-E3 are editing tasks graded by building and running

tests (E1: add an operator to a MASM program; E2: fix two bugs in a Python module; E3: add a CLI option to spec).

| Task | R2-mix | R1-mix | jakeatx ATX-IQ4_XS-M | Unsloth UD-IQ4_XS | vmarcelo IQ4-MIX | Ollama Q4_K_M |

|---|---|---|---|---|---|---|

| K1 (7 facts) | 7 | 6 | 7 | 7 | 5 | 7 |

| K2 (9 facts) | 9 | 9 | 9 | 9 | 7 | 7 |

| K3-K7 (17 facts) | 17 | 17 | 17 | 17 | 17 | 17 |

| K8 (6 facts) | 6 | 6 | 0 (hit the 10-min cap) | 6 | 6 | 6 |

| E1-E3 (3 tasks) | 3 | 3 | 3 | 3 | 3 | 3 |

| Tasks fully solved | 11/11 | 10/11 | 10/11 | 11/11 | 9/11 | 10/11 |

| Total time | 9.1 min | 9.5 min | 21.6 min | 10.8 min | 7.7 min | 9.5 min |

What this does and does not show: with one run per task, a difference of one task (or a single run that thinks

until the cap, as jakeatx did on K8) is within run-to-run noise, so R2-mix, R1-mix, jakeatx, UD-IQ4_XS and the

Ollama Q4_K_M should be read as roughly equal on this suite. The consistent signal is that the two quants with the

highest KLD (vmarcelo IQ4-MIX, Ollama Q4_K_M) are the ones that lost facts on K2. The tasks come from one workspace

and one domain. E1 for R2-mix and R1-mix was re-run after a prompt fix (the original prompt suggested `cmd /c

build.cmd`, which hangs under Qwen Code's Git Bash shell on Windows); the other quants ran with the fixed prompt only.

Speed and memory at 128k (RX 7900 XTX)

Setup: Windows 11, RX 7900 XTX 24 GB, Ryzen 7 7800X3D, 32 GB RAM; llama.cpp build 11558 (c83f3058) with ROCm 10.2.0a

(lemonade-sdk/llamacpp-rocm b1343). Workload: a three-turn conversation

(26.5k-token first prompt, then 5.7k and 5.2k new tokens per turn, reusing the cache), like an agent loop.

| Configuration | Prompt tok/s (turn 1 / 2 / 3) | Generation tok/s | Dedicated VRAM |

|---|---|---|---|

| R2-mix, 128k, KV q8_0, batch 2048 | 1,035 / 853 / 812 | 38.1-39.5 | 19.7 GB |

| R2-mix, 128k, KV q4_0, batch 2048 | 1,040 / 856 / 802 | 38.0-39.4 | 17.7 GB |

| R1-mix, 128k, KV q4_0, batch 2048 | 998 / 836 / 800 | 35.3-36.4 | 19.2 GB |

| Ollama library Q4_K_M via Ollama 0.40.3 (96k, its defaults) | 66 / 41 / 37 | 26-30 | spilled 3.7 GB into system RAM |

Long-context recall (three planted lines at 10/50/90% depth in ~58k and ~116k tokens of decompiler output): 3/3 at

both lengths for R2 with the 8-bit KV cache. The 4-bit cache was recall-tested only with the Ollama library Q4_K_M of the same

model (also 3/3 at both lengths), not with R2.

The speed and recall figures were measured on builds of these exact recipes made with a different imatrix (same tensor

types and file sizes, so the same memory traffic); the KLD table above is for the files in this repo.

Recommended llama-server command (24 GB GPU, 128k)

llama-server -m Qwen3.8-27B-R2-mix.gguf -ngl 99 -c 131072 -np 1 -fa on -b 2048 -ub 2048 \
  -ctk q8_0 -ctv q8_0 --cache-ram 2048 --ctx-checkpoints 8 --reasoning-effort medium \
  --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0

For R3-mix use -m Qwen3.8-27B-R3-mix.gguf -ctk q4_0 -ctv q4_0 -lm none with the same other flags (-lm none reads

the file once instead of memory-mapping it, keeping system RAM free).

Notes from tuning on Windows/ROCm:

• The embedded chat template defaults to reasoning_effort = xhigh when the request does not set one (Ollama sends

medium). At xhigh one code-review test spent its whole 8,192-token budget thinking without answering; at

medium it answered in 96 s. Set --reasoning-effort (or chat_template_kwargs) explicitly.

• On Windows a single process got about 20 GB of the card's 24 GB as dedicated VRAM; anything beyond silently spills to

shared system memory and prompt processing drops several-fold (4-5× with llama-server at 128k, about 15× in an

Ollama setup with MTP drafting in these tests). If other applications hold VRAM, use -ctk q4_0 -ctv q4_0.

• Large physical batches (-ub 2048) are much faster on ROCm than the default 512.
• MTP speculative decoding (--spec-type draft-mtp) needs a second context; at 98k (with the earlier Q4_K_M weights)

it spilled out of VRAM and slowed prompt processing about 8×. It may help at shorter contexts.

• If the system has an AMD CPU with integrated graphics, ROCm may list it as a second device; pin the GPU with -dev ROCm0.
• Recent llama.cpp applies a Hadamard rotation to quantized K caches automatically, which is why a 4-bit cache held up.

Recipe

Base type IQ4_XS; per-tensor overrides via llama-quantize --tensor-type-file (files in recipes/), imatrix from

unsloth/Qwen3.8-27B-GGUF (imatrix_unsloth.gguf), source

BF16/Qwen3.8-27B-BF16-*.gguf from the same repo.

| Tensors | R1 | R2 | R3 |

|---|---|---|---|

| ffn_up | IQ4_XS | IQ4_XS | Q5_K |

| ffn_gate | IQ4_XS | IQ4_XS | IQ4_XS |

| ffn_down | Q5_K | IQ4_XS | Q5_K |

| recurrent attn_qkv | Q5_K | Q5_K | Q5_K |

| recurrent attn_gate | Q4_K | IQ4_XS | Q5_K |

| recurrent ssm_out | Q5_K | IQ4_XS | Q5_K |

| full attention attn_q | Q5_K | Q4_K | Q5_K |

| full attention attn_output | Q6_K | Q5_K | Q6_K |

| full attention attn_k, attn_v | Q8_0 | Q8_0 | Q8_0 |

| ssm_alpha, ssm_beta | Q8_0 | Q8_0 | Q8_0 |

| output | Q6_K | Q5_K | Q5_K |

| token_embd | Q4_K | Q4_K | Q8_0 |

| MTP layer | kept | kept | removed |

llama-quantize --imatrix imatrix_unsloth.gguf --tensor-type-file recipes/R2.txt \
  --output-tensor-type q5_k --token-embedding-type q4_k Qwen3.8-27B-BF16-00001-of-00002.gguf out.gguf IQ4_XS

(R1: recipes/R1.txt, --output-tensor-type q6_k.) R3: recipes/R3.txt, `--output-tensor-type q5_k

--token-embedding-type q8_0 --prune-layers 64, then gguf-set-metadata out.gguf qwen35.nextn_predict_layers 0`

(without it llama.cpp still expects the removed MTP layer and runs 63 layers).

Related quants

The idea here (IQ4_XS for the bulk feed-forward weights, more bits for attention and the recurrent layers) is not

new; several earlier Qwen3.8-27B releases do the same or more:

• jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF: IQ4_XS base

with Q4_K_M-style upgrades, built for a 24 GB RTX 3090 at 200K context with an 8-bit K cache and MTP.

• vmarcelo/Qwen3.8-27B-MIX_GGUF: custom per-tensor mixes

with an imatrix, built from the FP8 checkpoint for 16 GB cards.

• utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF:

attention and DeltaNet projections at Q8_0 with NVFP4 feed-forward weights.

• tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:

automatic rate-distortion bit allocation with an imatrix.

Measured with the same KLD setup as above:

| Model | Size (MiB) | Mean KLD | Same top token | 99.9th pct KLD |

|---|---|---|---|---|

| R2-mix (this repo) | 14,677 | 0.0159 ± 0.0005 | 94.3% | 0.62 |

| jakeatx ATX-IQ4_XS-M | 14,866 | 0.0164 ± 0.0004 | 94.3% | 0.67 |

| vmarcelo IQ4-MIX | 13,458 | 0.0525 ± 0.0038 | 92.2% | 6.26 |

R2-mix and ATX-IQ4_XS-M are effectively tied. IQ4-MIX is smaller and was made from the FP8 checkpoint, while the

reference here is a Q8_0 of the BF16 weights, so part of its distance is the FP8 source rather than its recipe.

What this repo adds is mainly the measurements and the Windows/ROCm 128k configuration notes above.

License and credits

Apache 2.0, as the base model; see LICENSE. Base model by the Qwen team. Hosted-model comparison run through a

ChatGPT subscription; model names as listed by that service on 2026-10-12. BF16 GGUF conversion and imatrix by

Unsloth. Quantization recipe and measurements by jdominick05.