---

license: apache-2.0

base_model: Qwen/Qwen3.6-35B-A3B

tags:

- gguf

- imatrix

- llama.cpp

- quantized

- verified

- qwen

- qwen3.6

- moe

pipeline_tag: text-generation

---

🐋 Qwen3.6-35B-A3B — Imatrix GGUF

Verified quants only. Sub-4-bit variants are excluded from this release because they produce degenerate output at this bit-rate — no point shipping broken files.

GGUF imatrix builds of Qwen/Qwen3.6-35B-A3B — a Mixture-of-Experts model with 35B total parameters but only ~3B active per token, plus hybrid attention/state-space (SSM) layers. Flagship-level reasoning, coding and multilingual RAG at small-model speeds.

Built with llama.cpp and importance-matrix calibration on a public multilingual + code + math corpus. Every quant is loaded, prompted, and visually checked before publication.

👋 From the author

Hi local LLM enthusiasts! You may have noticed a few one-off quants from me over the past months. This release comes from a fully automated pipeline — same as my Gemma 4 line.

Please try them out and let me know which quants are useful or which ones you wish I'd made. Anything below 4-bit produces degraded output on this architecture, so I'm holding off on those until they actually work well. More quants and more models coming.

🎯 Pick a quant

| Quant | Size | What it's for |

|-------|------|---------------|

| Q8_0 | 34G | Almost the original. Pick this if RAM isn't a concern. |

| Q6_K_L | 27G | Near-lossless with Q8_0 embeddings. Best of the K-quants. |

| Q6_K | 27G | Near-lossless. Excellent fidelity at a smaller size than Q8_0. |

| Q5_K_L | 23G | Q5_K_M with Q8_0 embeddings. High quality, small overhead. |

| Q5_K_M | 23G | Sweet spot between Q4 and Q6. Solid all-rounder. |

| Q5_K_S | 22G | Slightly smaller than Q5_K_M, virtually identical output. |

| Q4_K_L | 20G | Q4_K_M with Q8_0 embeddings. The smart 4-bit choice. |

| Q4_K_M | 20G | The default. Best size/quality tradeoff. |

| Q4_K_S | 19G | A bit smaller than Q4_K_M, a tiny step down. |

| IQ4_NL | 20G | ARM-optimized 4-bit. For Raspberry Pi & friends. |

| IQ4_XS | 18G | Tightest 4-bit format. Quality close to Q4_K_S, smaller. |

_L variants override the output tensor and embeddings to Q8_0 — small disk cost, better output stability.*

Tl;dr:

• 🟢 Q4_K_M — start here unless you have a reason not to
• 🟡 Q5_K_M — small quality bump if you have RAM to spare
• 🔵 Q8_0 — pick this if you're not RAM-constrained and want max quality
• 🟠 IQ4_XS — when 4-bit is too big
• 🟣 IQ4_NL — running on ARM (Pi, phones)

🎯 Use cases tested

This release has been verified against real workloads, not just a smoke prompt:

• Cross-language RAG (EN ↔ RU) — bridging English data-center / infrastructure documentation with Russian situational queries, retrieval grounding preserved across languages.
• Algorithmic coding — generating O(N)-complexity Python without regex, while strictly following constraint rules.
• Strict formatting — emitting pure, valid JSON without markdown wrappers or conversational filler.

💬 Prompt format

ChatML:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

📥 Download

Single file (recommended):

hf download Krasnopjorovs/Qwen3.6-35B-A3B-Imatrix-GGUF \
  --include "Qwen3.6-35B-A3B-Q4_K_M.gguf" --local-dir .

Whole repo:

hf download Krasnopjorovs/Qwen3.6-35B-A3B-Imatrix-GGUF --local-dir ./qwen3.6-35b-a3b-gguf

▶ Run

./llama-server \
  -m Qwen3.6-35B-A3B-Q4_K_M.gguf \
  -c 32768 -ngl 99 \
  --host 0.0.0.0 --port 8080

Then point your favorite chat client at http://localhost:8080/v1.

MoE with only ~3B active parameters means generation is fast even when the full weights are large — a 4-bit build fits comfortably on a 24 GB GPU with room for a long context window.

🔬 Calibration

Imatrix generated from reapmix — a community calibration mix (multilingual + code + math, ~305K words / ~4.4 MB). Same class of public calibration data used by other community publishers; this release makes no unique calibration claim.

The output tensor and embeddings carry disproportionate weight in quantized output. The *_L variants keep these at Q8_0 — small disk cost for noticeably better stability at low bit-rates.

✅ Build info

• Source: Qwen/Qwen3.6-35B-A3B
• Architecture: qwen35moe (MoE + hybrid SSM/attention)
• llama.cpp: latest mainline
• Quantization: CPU
• Imatrix calibration: GPU
• Quant whitelist: 4-bit and above only

🙏 Credits

• Source model — Qwen (Apache 2.0)
• Calibration corpus — reapmix by eaddario
• Quantization tooling — llama.cpp

---

built · © 2026 Krasnopjorovs