---

license: apache-2.0

base_model: Qwen/Qwen3.6-27B

tags:

• gguf
• text-generation
• qwen
• ik_llama.cpp
• nvidia
• imatrix
• 4-bit
• iq4_ks

pipeline_tag: text-generation

quantized_by: cHunter789

---

Qwen3.6-27B-i1-IQ4_KS-GGUF

This repository contains GGUF format weights for the Qwen3.6-27B model, quantized using the ik_llama.cpp project.

This model was specifically created to run on consumer GPUs with 16GB VRAM. By utilizing q4_0 KV cache quantization, it allows pushing the context length up to 105k tokens.

> ⚠️ Note: This model is designed exclusively for nVidia GPUs and is based on the advanced KS quants developed by ikawrakow from the ik_llama.cpp repository.

!!!UPDATE!!!

I added a new quantization version which keeps the same file size but moves the model's weight from knowledge layers to logic layers. This results in a slight hit to PPL but should improve the model's performance on coding tasks. The new version is Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS. I also added a version with an MTP head.

---

Quantization Details & Imatrix File

• Quantization Base: KS Quants (ik_llama.cpp).
• Imatrix File Used: Provided by mradermacher.
• Other Tested Imatrix Files:

bartowski* – yielded significantly worse results.

ubergarm* – yielded comparable results.

* _If you find or generate a better Imatrix file, please let me know in the Community tab!_

I have also included the script I used for the quantization process in this repository. If you have any concerns regarding malware or security, you can download the official BF16 weights from Unsloth and quantize the model yourself using the provided script.

<details>

<summary><b>secret recipe Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS (test11)</b></summary>

# 1. Base Tensors & Embeddings
^output\.weight$ -> iq5_k
^output_norm\.weight$ -> f32
^token_embd\.weight$ -> iq4_ks

# 2. Exceptions (High precision for the first block's attention gate)
^blk\.0\.attn_gate\.weight$ -> iq6_k

# 3. Feed-Forward Networks (FFN)
# Aggressive compression (iq4_kss) for middle blocks 5-51
^blk\.([5-9]|[1-4][0-9]|50|51)\.ffn_(down|up)\.weight$ -> iq4_kss
.*ffn_(down|up)\.weight$ -> iq4_ks
.*ffn_gate\.weight$ -> iq4_ks

# 4. Attention Layers
.*attn_v\.weight$ -> iq5_ks
.*attn_(qkv|q|k|output)\.weight$ -> iq4_kss
.*attn_gate\.weight$ -> iq4_ks

# 5. Mamba / SSM Layers
.*ssm_out\.weight$ -> iq4_kss
.*ssm_(alpha|beta)\.weight$ -> iq4_ks
.*ssm_conv1d\.weight$ -> f32
.*ssm_a$ -> f32
.*ssm_dt\.bias$ -> f32

# 6. All Normalization Layers (Kept in f32 for stability)
.*norm\.weight$ -> f32

</details>

<details>

<summary><b>secret recipe Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS (test29)</b></summary>

# Hybrid Sandwich quantization — from quantize_hybrid_sandwich_logic_test29.sh
# Pattern: alternating fused QKV (with SSM) / split QKV (no SSM) blocks
#   - Blocks n%4 == 0,1,2 (n<60): fused QKV + SSM
#   - Blocks n%4 == 3:            split Q (iq4_kss) / K / V (iq5_ks) / output — no SSM
#   - Blocks 60-62: fused QKV + SSM (no iq4_kss FFN)
#   - Blocks 63:  split QKV — no SSM

# 1. Base Tensors & Embeddings
^output\.weight$ -> iq5_k
^output_norm\.weight$ -> f32
^token_embd\.weight$ -> iq4_ks

# 2. Exceptions — first blocks & tail blocks (higher precision gates)
^blk\.0\.attn_gate\.weight$ -> q8_0
^blk\.([1-2]|6[01])\.attn_gate\.weight$ -> iq6_k
^blk\.62\.attn_gate\.weight$ -> q8_0

# 3. Feed-Forward Networks (FFN)
# Aggressive compression (iq4_kss) for middle blocks 4-59
^blk\.([4-9]|[1-5][0-9])\.ffn_(down|up)\.weight$ -> iq4_kss
.*ffn_(down|up)\.weight$ -> iq4_ks
.*ffn_gate\.weight$ -> iq4_ks

# 4. Attention Layers
# 4a. Fused QKV blocks: 0-2, 4,8,12..56, 60-62
^blk\.(([0-2])|([0-4][048])|(5[26])|(6[0-2]))\.attn_qkv\.weight$ -> iq4_ks

# 4b. Split QKV blocks: 3,7,11,15..59,63
^blk\.(([37])|([1-5][13579])|(63))\.attn_q\.weight$ -> iq4_kss
^blk\.(([37])|([1-5][13579])|(63))\.attn_k\.weight$ -> iq4_ks
^blk\.(([37])|([1-5][13579])|(63))\.attn_v\.weight$ -> iq5_ks
^blk\.(([37])|([1-5][13579])|(63))\.attn_output\.weight$ -> iq4_ks

# 4c. Default attention gate
.*attn_gate\.weight$ -> iq4_ks

# 5. Mamba / SSM Layers (only in fused QKV blocks)
.*ssm_out\.weight$ -> iq4_ks
.*ssm_(alpha|beta)\.weight$ -> iq4_ks
.*ssm_conv1d\.weight$ -> f32
.*ssm_a$ -> f32
.*ssm_dt\.bias$ -> f32

# 6. All Normalization Layers (Kept in f32 for stability)
.*norm\.weight$ -> f32

</details>

Quantization Recipe Comparison

| Aspect | Original (test11) | New (test29) |

|---|---|---|

| Block 0 attn_gate | iq6_k | q8_0 |

| Blocks 1-2 attn_gate | iq6_k | iq6_k |

| FFN iq4_kss range | blocks 5-51 | blocks 4-59 |

| Attention pattern | uniform iq4_kss for all qkv/q/k | alternating fused/split per block |

| Fused QKV blocks | — | iq4_ks (blocks 0-2, 4,8,12..56, 60-62) |

| Split Q blocks | iq4_kss (all) | iq4_kss (blocks 3,7,11..59,63 only) |

| Split V blocks | iq5_ks (all) | iq5_ks (blocks 3,7,11..59,63 only) |

| SSM ssm_out | iq4_kss | iq4_ks |

| Blocks 60-62 | not specified | fused QKV + SSM, iq4_ks FFN |

| Block 62 attn_gate | — | q8_0 |

| Blocks 60-61 attn_gate | — | iq6_k |

Benchmark Results & Real-World Impressions

The model was heavily tested in daily production workflows for several days. It runs much faster (1.5x-1.75x) and more reliably than the previous iteration—completely eliminating the issue of "blank outputs", while the search-replace functionality works flawlessly.

• Qwen Benchmark: Successfully passed the performance evaluations on qwen3-6-27b-benchmark.vercel.app.
• Needle In A Haystack: Successfully evaluated with satisfying results across the full 100k context window.
• Comparison: In direct testing, this model performs slightly better than my previous variant: Qwen3.6-27B-i1-IQ4_XS-GGUF.

Perplexity (PPL) — Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS (test11)

Perplexity evaluations were conducted focusing exclusively on the KV Cache quantization setup (q4_0), as this is the primary target use case:

wget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt \

./llama-perplexity -m Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS.gguf -f pg19.txt -c 65536 --chunks 32 -ngl 99 -khad -vhad -ctk q4_0 -ctv q4_0 -fa 1 -b 512 -ub 512

Test Log Output:

perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 71.10 seconds per pass - ETA 14.22 minutes
[1]6.6897,[2]7.0032,[3]7.1989,[4]7.3327,[5]7.4816,[6]7.3770,[7]7.4325,[8]7.4378,[9]7.4754,[10]7.5192,[11]7.5669,[12]7.4040,

Final estimate: PPL over 12 chunks for n_ctx=65536 = 7.4040 +/- 0.02773

Perplexity (PPL) — Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS (test29)

./llama-perplexity -m Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KS.gguf -f /mnt/Samsung4TB/models/pg19.txt -c 65536 --chunks 32 -ngl 99 -khad -vhad -ctk q4_0 -ctv q4_0 -fa 1 -b 512 -ub 256

Test Log Output:

[1]6.6926,[2]7.0049,[3]7.2043,[4]7.3382,[5]7.4861,[6]7.3838,[7]7.4411,[8]7.4459,[9]7.4857,[10]7.5303,[11]7.5779,[12]7.4131,

Final estimate: PPL over 12 chunks for n_ctx=65536 = 7.4131 +/- 0.02774

PPL Comparison:

| Test | Recipe | PPL (n_ctx=65536) | Margin of Error |

|---|---|---|---|

| test11 | Original v1 (attn_qkv-IQ4_KSS) | 7.4040 | +/- 0.02773 |

| test29 | Hybrid Sandwich v2 | 7.4131 | +/- 0.02774 |

The difference of ~0.009 is well within the margin of error, indicating statistically equivalent perplexity between both quantization strategies.

---

Production Deployment Configuration (ik_llama.cpp)

Below is the verified and stable execution configuration used for llama-server. It utilizes the corrected chat template (chat.template) by ex-arman68, sourced from froggeric/Qwen-Fixed-Chat-Templates.

llama-server \
        -m "$MODEL_PATH" \
        -a Qwen3.6-27B \
        --ctx-size 105000 \
        --chat-template-file chat_template.jinja \
        --n-gpu-layers 99 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --batch-size 512 \
        --ubatch-size 256 \
        --flash-attn on \
        --no-mmap \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format deepseek \
        -t 8 \
        --parallel 1 \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true}' \
        --defrag-thold 0.3 \
        --jinja \
        --cont-batching \
        --temp 0.15 \
        --top-k 1 \
        --min-p 0.1 \
        --keep -1 \
        --repeat-last-n 512 \
        --repeat-penalty 1.05

Versions of the model with MTP have also been prepared. i1_MTP with iq4_ks quants and MPT with q8_0. Here is an example command to run the model with MTP:

llama-server \
        -m "$MODEL_PATH" \
        -a Qwen3.6-27B \
        --ctx-size 50000 \
        --chat-template-file chat_template.jinja \
        --n-gpu-layers 99 \
        -mtp \
        --spec-type mtp \
        --draft-n 3 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --batch-size 512 \
        --ubatch-size 256 \
        --flash-attn on \
        --no-mmap \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format deepseek \
        -t 8 \
        --parallel 1 \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true}' \
        --defrag-thold 0.3 \
        --jinja \
        --cont-batching \
        --temp 0.15 \
        --top-k 1 \
        --min-p 0.1 \
        --keep -1 \
        --repeat-last-n 512 \
        --repeat-penalty 1.05