steampunque/Qwen3-Coder-30B-A3B-Instruct-MP-GGUF overview

---

license: apache-2.0

base_model: Qwen/Qwen3-Coder-30B-A3B-Instruct

base_model_relation: quantized

tags:

• Qwen
• Qwen3 Coder
• GGUF
• quantized
• 4-bit

---

Mixed Precision GGUF layer quantization of Qwen3-Coder-30B-A3B-Instruct by Qwen

Original model: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct

The hybrid quant employs different quantization levels on a per layer basis to increase

flexibility of trading off performance vs file size. Less parameter bits are used at deep layers

and more bits at cortex layers to simulultaneously optimize quantized size and model performance.

An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 9/18/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_886"],[1 ,"Q5_K_885"],[2 ,"Q5_K_664"],[3 ,"Q4_K_664"],[4 ,"Q4_K_654"],[5 ,"Q4_K_554"],[6 ,"Q4_K_554"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_444"],[9 ,"Q4_K_444"],[10,"Q4_K_444"],[11,"Q4_K_444"],[12,"Q4_K_444"],[13,"Q4_K_444"],[14,"Q4_K_444"],[15,"Q4_K_444"],
   [16,"Q4_K_554"],[17,"Q4_K_554"],[18,"Q4_K_554"],[19,"Q4_K_554"],[20,"Q4_K_654"],[21,"Q4_K_654"],[22,"Q4_K_654"],[23,"Q4_K_654"],
   [24,"Q4_K_654"],[25,"Q4_K_654"],[26,"Q4_K_654"],[27,"Q4_K_654"],[28,"Q4_K_654"],[29,"Q4_K_654"],[30,"Q4_K_654"],[31,"Q4_K_654"],
   [32,"Q4_K_664"],[33,"Q4_K_664"],[34,"Q4_K_664"],[35,"Q4_K_664"],[36,"Q4_K_664"],[37,"Q4_K_664"],[38,"Q4_K_664"],[39,"Q4_K_664"],
   [40,"Q4_K_665"],[41,"Q4_K_665"],[42,"Q4_K_665"],[43,"Q5_K_665"],[44,"Q5_K_665"],[45,"Q5_K_666"],[46,"Q5_K_668"],[47,"Q6_K_866"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized for strong performance across a small set of code gen prompts on both cuda backend

and CPU backend with CPU expert offload while sized to approximately Q4_K_M bit efficiency.

Comparison:

Quant | size | PPL | Comment

---------|---------|------|-----------

Q4_K_M | 18.6e9 | 9.6 | default embed and output

Q4_E_H | 18.3e9 | 9.4 | Q6_K embed Q6_K output

Usage:

This moe model can be efficiently run by offloading expert tensors to CPU via -ot exps=CPU

to open up very large context space. The smaller size of the optimally quantized parameters will give

an effective boost in CPU processing speed due to reducing the memory BW needed to repeatedly copy them

from main memory to SIMD regs. It can also run fully offloaded on GPU via RPC or high VRAM GPU.

The model can be speculated using Qwen2.5-Coder-0.5B-Instruct if the inference platform can support

vocabulary translation between draft and target. Example performance on humaneval first problem prompt using

a downstream speculator with fixed draft block length ND:

Config | ND | F16 context size | Q8 context size | gen rate

---------------------|--------|-------------------|-----------------|---------

2 4070, RPC | 0 | ~55k tokens | ~100k tokens | 112tps

2 4070, RPC | 4 | ~45k tokens | ~75k tokens | 142tps

1 4070, exp CPU | 0 | ~105k tokens | ~200k tokens | 22 tps

1 4070, exp CPU | 2 | ~90k tokens | ~155k tokens | 25 tps

The model has a native 256k context window which can be extended to 1M with Yarn as follows:

Set base context for yarn rope scale compute to 262144 (256k), then with a context of N tokens > 256k the rope scale = N / 256k.

Example, if N = 512k then rope scale = 2.0.

Then on model start pass --rope-scaling yarn --yarn-orig-ctx 262144 --rope_scale 2.0 (must be ajusted if kv other than 512k)

Later versions of llama.cpp have a bug which soft caps context length to the training context, effectively disabling yarn context extension. Patch server-context.cpp according to https://github.com/ggml-org/llama.cpp/issues/22140 to fix it.

High context performance appears to work verified against a needle in haystack test at 75k tokens.

Benchmarks:

Code evals for the model are given here: https://huggingface.co/spaces/steampunque/benchlm.

Download the file from below:

| Link | Type | Size/e9 B | Notes |

|------|------|-----------|-------|

| Qwen3-30B-Coder-A3B-Instruct.Q4_E_H.gguf | Q4_E_H | 18.3e9 B | ~Q4_K_M size |

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040