steampunque/Qwen3.5-27B-MP-GGUF overview

---

license: apache-2.0

base_model: Qwen/Qwen3.5-27B

base_model_relation: quantized

tags:

• Qwen3.5 27B
• GGUF
• quantized
• 4-bit
• 6-bit
• mixed precision

---

Mixed Precision GGUF layer quantization of Qwen3.5-27B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.5-27B

The hybrid quant employs different quantization levels on a per layer basis to enable

both high performance and small file size at the same time. The quants employed are all K to avoid

slow CPU or older GPU processing of IQ quants. An extended layer definition E quant Q4_E_H for the model

is defined as follows (updated 8/25/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOSD"],
   [0 ,"Q5_K_6666"],[1 ,"Q5_K_6665"],[2 ,"Q4_K_6555"],[3 ,"Q4_K_6555"],
   [4 ,"Q4_K_5555"],[5 ,"Q4_K_5555"],[6 ,"Q4_K_4444"],[7 ,"Q4_K_5554"],
   [8 ,"Q4_K_4444"],[9 ,"Q4_K_4444"],[10,"Q4_K_4444"],[11,"Q4_K_5554"],
   [12,"Q4_K_4444"],[13,"Q4_K_4444"],[14,"Q4_K_4444"],[15,"Q4_K_5554"],
   [16,"Q4_K_4444"],[17,"Q4_K_4444"],[18,"Q4_K_4444"],[19,"Q4_K_5554"],
   [20,"Q4_K_4444"],[21,"Q4_K_4444"],[22,"Q4_K_4444"],[23,"Q4_K_5554"],
   [24,"Q4_K_4444"],[25,"Q4_K_4444"],[26,"Q4_K_4444"],[27,"Q4_K_5554"],
   [28,"Q4_K_5444"],[29,"Q4_K_5444"],[30,"Q4_K_5444"],[31,"Q4_K_5554"],
   [32,"Q4_K_5454"],[33,"Q4_K_5454"],[34,"Q4_K_5454"],[35,"Q4_K_5555"],
   [36,"Q4_K_5454"],[37,"Q4_K_5454"],[38,"Q4_K_5454"],[39,"Q4_K_5555"],
   [40,"Q4_K_5554"],[41,"Q4_K_5554"],[42,"Q4_K_5554"],[43,"Q4_K_5555"],
   [44,"Q4_K_5554"],[45,"Q4_K_5554"],[46,"Q4_K_5554"],[47,"Q4_K_5555"],
   [48,"Q4_K_5554"],[49,"Q4_K_5554"],[50,"Q4_K_5554"],[51,"Q4_K_5555"],
   [52,"Q4_K_5555"],[53,"Q4_K_5555"],[54,"Q4_K_5555"],[55,"Q4_K_6665"],
   [56,"Q4_K_5555"],[57,"Q4_K_5555"],[58,"Q4_K_5555"],[59,"Q5_K_5565"],
   [60,"Q5_K_6665"],[61,"Q5_K_6666"],[62,"Q5_K_6668"],[63,"Q6_K_8666"],
   [64,"Q4_K_6554"]
   ]'
   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high"

The quant was sized at ~Q4_K_M bpw and optimized for strong performance over a small set of curated reasoning prompts.

Comparison:

Quant | size | PPL | Comment

-------|---------|-------|-----------

Q4_K_M | 16.8e9 | 8.3 | Q4_K_M with default embedding and output

Q4_E_H | 16.7e9 | 8.5 | Hybrid quant with Q4_K embedding Q6_K output (includes 0.3B MTP layer)

Usage:

Qwen3.5-27B is a vision capable dense RL model. It can be used together with its multimedia projector layers to process images and text inputs

and generate text outputs. The mmproj file is made available in this repository.

Due to the attention scheme used in the model, straightforward speculation approaches cannot be used.

Later versions of llama.cpp as of 4/23/2026 attempt to circumvent this problem however efficiency will

not be good so use of speculation prior to Qwen3.5 MTP patches is not recommended.

As of llama.cpp b9180 MTP support for the model was added to upstream and may be experimented with by using the Q4_E_H_MTP quant.

On a 2x 4070 setup (1 RPC) approx performance is:

Q | QKV | NKV | gen tps

-------|----------|--------|--------

Q4_E_H | F16 | 100k +| 25

Q4_E_H | Q8_0 | 160k +| 23

The model appears to be trained to decide itself whether to do a think block or not. When it does a think block it can fall into very

heavy overthinking on ambiguous/trick prompts. Over a small set of eval prompts the model did extremely well. To avoid

the overthinking inject think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into

a think block inject a bootstrap think start following the assistant prompt:

"<think>\n"

The model was found to be highly capable on reasoning tasks when skipping think block.

VISION:

The model was tested in vision mode on a couple pretty tough bird ID image and did extremely well handling the images with concise,

informative, accurate, and efficient reasoning blocks.

CODE:

The model was tested across a small set of code gen prompts and found to be quite capable in its ability to generate working code, though

it did fall into a rep loop with think enabled on one of the test prompts.

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

| Link | Type | Size/e9 B | Notes |

|------|------|-----------|-------|

| Qwen3.5-27B.Q4_E_H.gguf | Q4_E_H | 16.7e9 B | ~Q4_K_M bpw, includes 0.3B MPT layer |

| Qwen3.5-27B.mmproj.gguf | F16 | 0.93e9 B | multimedia projector |

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040