---

license: apache-2.0

base_model: Qwen/Qwen3.8-27B

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- qwen3.5

- gguf

- llama.cpp

- imatrix

- text-only

- quantized

---

Qwen3.8-27B · Text-Only · GGUF (imatrix)

**Qwen3.8-27B for llama.cpp — vision tower and MTP head removed at the weight level,

importance-matrix quantized from the original BF16. Runs from 16 GB VRAM up.**

Optional vision is available as a separate mmproj file (see below), so you only

pay for it if you use it.

> Community quantization. Not an official Qwen release; not endorsed by or

> affiliated with the Qwen team or Alibaba Cloud. "Qwen3.8" identifies the upstream

> model this artifact derives from (Apache-2.0 §6).

---

Which file do I download?

| File | Size | pp512 | tg128 | Max context on one 3090 | |

|---|---|---|---|---|---|

| Q8_0 | 26.6 GiB | — | — | does not fit | Near-lossless. Needs 32 GB, or partial offload on 24 GB |

| Q6_K | 20.6 GiB | 1088 | 32.3 | 32,512 | Best quality on one 24 GB card — but look at that context |

| Q5_K_M | 17.9 GiB | 1110 | 36.9 | 73,472 | |

| Q4_K_M | 15.4 GiB | 1183 | 41.4 | 111,872 | Fastest, and 3.4× the context of Q6_K. The default pick |

| IQ4_XS | 14.0 GiB | 1042 | 31.9 | 133,632 | Pick for a 16 GB card — see the caveat below |

| Q3_K_M | 12.4 GiB | 992 | 38.5 | 158,976 | Quality drops noticeably |

| mmproj-* | 0.9 GiB | — | — | — | Optional vision, pairs with any of the above |

All three columns measured on one RTX 3090 (24 GB), Vulkan, full offload. Context

figures come from llama-fit-params, i.e. what actually fits — not a calculation.

The size/context trade is steeper than the size/quality trade. Dropping from Q6_K

to Q4_K_M costs a little quality and buys 3.4× the context — on the same card.

The counter-intuitive part: IQ4_XS is 9 % smaller than Q4_K_M but ~23 % slower.

IQ-family quants cost more compute to dequantize. Take IQ4_XS because you need the

size, not because you want speed. If Q4_K_M fits your card, it is both faster and

higher quality.

16 GB card: IQ4_XS. Q4_K_M technically loads but leaves almost nothing for KV.

24 GB card: Q4_K_M for speed, Q6_K for quality.

Apple Silicon: unified memory is the budget — 32 GB → Q5_K_M/Q6_K, 64 GB → Q8_0.

Vulkan numbers. CUDA builds are typically faster.

---

Why the KV cache is unusually small

Qwen3.8 is a hybrid-attention model. Of its 64 layers only 16 are full attention —

the other 48 are Gated DeltaNet linear attention and hold no KV cache. With

num_key_value_heads = 4, head_dim = 256:

| KV dtype | Per token | 32K ctx | 128K ctx |

|---|---|---|---|

| f16 | 64 KiB | 2.0 GiB | 8.0 GiB |

| q8 | 32 KiB | 1.0 GiB | 4.0 GiB |

A comparable dense-attention 27B needs roughly four times this — which is why the

context figures in the table above are as large as they are.

*The table is measured on a 24 GB card. For a 16 GB card with IQ4_XS (14.0 GiB) the

same arithmetic gives roughly 32K at f16 KV or 64K at q8 — estimated, not measured,

as the author has no 16 GB card to test on.*

---

Vision

llama-mtmd-cli -m Qwen3.8-27B-Text-Only-Q4_K_M.gguf \
               --mmproj mmproj-Qwen3.8-27B-f16.gguf \
               --image photo.jpg -p "Describe this image."

The projector is the general-purpose tower from Qwen3.8-27B, unchanged. Dedicated

Qwen3-VL-* models will still do better on dense OCR and small-object counting.

---

How these were made

Quantized from the original BF16 weights, not re-quantized from an existing INT4

release — so no compounding loss.

1. Vision tower (model.visual.) and MTP block (model.mtp) dropped at the

safetensors level; model.language_model. promoted to model.

1. convert_hf_to_gguf.py --outtype bf16 --no-mtp → BF16 GGUF (lossless from source)
2. Importance matrix over 300 chunks of the same calibration corpus used for this

author's W4A16 releases (512 passages, pile-val news text), on 2×RTX 3090

1. Every level quantized with that imatrix, K-quants included

Every file in this repo was loaded and generated with before publishing — including

both mmproj files, which were checked against a synthetic image with known content.

Not just checksum-verified.

---

Requirements

A llama.cpp build that knows the qwen35 architecture. Build b10502 or newer works.

> Older builds fail with check_tensor_dims: tensor 'blk.64...' not found. That is a

> converter that counted an MTP block these files do not contain — not a corrupt download.