---

license: apache-2.0

language:

• en

base_model:

• Qwen/Qwen3.6-35B-A3B

pipeline_tag: text-generation

tags:

• gguf
• quantized
• qwen
• qwen3.6
• Qwen3.6-35B-A3B
• moe
• mixture-of-experts
• 2-bit
• 4-bit
• iq2_xxs
• q2_k
• q4_k
• q8_0
• q36
• quarkstar
• vulkan
• antirez
• radv
• bc-250
• conversational

---

Qwen3.6-35B-A3B — GGUF for QuarkStar

These GGUF weights are built for QuarkStar, a small native inference engine specialized for Qwen3.6-35B-A3B.

The quantization recipe is inspired by Salvatore Sanfilippo's asymmetric DeepSeek-V4 quantization work in DS4: aggressively quantize the routed experts, which contain most of the parameters, while keeping routing, shared experts, attention, recurrent blocks, embeddings, and output tensors at higher precision.

The result is a roughly 11 GB model capable of fitting fully resident on a 16 GB unified-memory machine, with enough memory left for the runtime, KV cache, and operating system.

Files

| File | Size | Purpose |

| ------------------------------------------------------------------------------------------------- | ------: | -------------------------------------------------------------------- |

| Qwen3.6-35B-A3B-AntirezExperts-IQ2XXS-gateup-Q2K-down-Q8rest.gguf | 11.7 GB | Default Q2 model for fully resident use on a 16 GB BC-250 |

| Qwen3.6-35B-A3B-Layers34-39Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-Q8Rest-imatrix.gguf | 13.1 GB | Higher-quality mixed Q2/Q4 model; SSD streaming recommended on 16 GB |

| Qwen3.6-35B-A3B-MTP-Q4K-Q8_0-F32.gguf | 1.59 GB | Optional MTP model for experimental speculative decoding |

The MTP file is not a standalone language model. Pair it with either main model and enable it explicitly with --mtp.

Which model should I use?

16 GB unified memory

Use the default Q2 model:

./download_model.sh q2-imatrix

It is designed to fit fully resident on 16 GB while leaving memory for the operating system, scratch buffers and KV cache.

Higher quality

Use the mixed Q2/Q4 model:

./download_model.sh q2-q4-imatrix

The last six transformer layers, layers 34 through 39, use Q4_K routed experts.

On a 16 GB, run this model with SSD streaming:

./q36 \
  -m gguf/Qwen3.6-35B-A3B-Layers34-39Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-Q8Rest-imatrix.gguf \
  --ssd-streaming \
  -p "Hello"

Optional MTP

Download the MTP component separately:

./download_model.sh mtp

Enable it explicitly:

./q36 \
  --mtp gguf/Qwen3.6-35B-A3B-MTP-Q4K-Q8_0-F32.gguf \
  --mtp-draft 2 \
  -p "Hello"

The current MTP path is experimental and generally provides only a slight speedup.

Quantization recipe

Default Q2 quantization recipe

File:

Qwen3.6-35B-A3B-AntirezExperts-IQ2XXS-gateup-Q2K-down-Q8rest.gguf

| Tensor class | Quantization |

| -------------------------------------- | ------------- |

| blk.*.ffn_gate_exps.weight | IQ2_XXS |

| blk.*.ffn_up_exps.weight | IQ2_XXS |

| blk.*.ffn_down_exps.weight | Q2_K |

| Shared experts | Mostly Q8_0 |

| Attention projections | Mostly Q8_0 |

| Recurrent and SSM matrices | Mostly Q8_0 |

| Token embeddings | Q8_0 |

| Output head | Q8_0 |

| Norms, gates, biases and small tensors | F32 |

Only the routed MoE expert tensors are changed from the source Q8 GGUF.

Tensor counts

| Type | Count |

| --------- | ------: |

| F32 | 301 |

| Q8_0 | 312 |

| IQ2_XXS | 80 |

| Q2_K | 40 |

| Total | 733 |

Conversion summary

80 routed gate/up tensors -> IQ2_XXS
40 routed down tensors    -> Q2_K
all other tensors         -> unchanged

| Property | Value |

| --------------- | -------------------: |

| Source size | 36,903,140,320 bytes |

| Final size | 11,737,316,384 bytes |

| Changed tensors | 120 |

| Imatrix entries | 120 |

| Imatrix chunks | 6,249 |

Mixed Q2/Q4 quantization recipe

File:

Qwen3.6-35B-A3B-Layers34-39Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-Q8Rest-imatrix.gguf

Layers 0 through 33 use the default asymmetric Q2 expert recipe:

| Tensor class | Quantization |

| ------------ | ------------ |

| Gate experts | IQ2_XXS |

| Up experts | IQ2_XXS |

| Down experts | Q2_K |

Layers 34 through 39 use Q4_K for all three routed-expert matrices:

| Tensor class | Quantization |

| ------------ | ------------ |

| Gate experts | Q4_K |

| Up experts | Q4_K |

| Down experts | Q4_K |

Everything outside the routed experts retains the same higher-precision source type.

Tensor counts

| Type | Count |

| --------- | ------: |

| F32 | 301 |

| Q8_0 | 312 |

| IQ2_XXS | 68 |

| Q2_K | 34 |

| Q4_K | 18 |

| Total | 733 |

Usage

git clone https://github.com/Ninnix/q36
cd q36
./download_model.sh q2-imatrix     # 16 GB RAM machines
./download_model.sh q2-q4-imatrix     # >= 24 GB or enable ssd-streaming on 16 GB 
./download_model.sh mtp    # optional MTP / speculative decoding
make

./q36 -p "Explain Redis streams in one paragraph."
./q36-server --ctx 32000 --kv-disk-dir /tmp/q36-kv --kv-disk-space-mb 8192

The download_model.sh script fetches from this repository, resumes partial downloads, and points ./q36moe.gguf at the selected variant.

Acknowledgements

The asymmetric expert-quantization strategy and much of the runtime direction are inspired by Salvatore Sanfilippo's work on:

• DS4

QuarkStar is a Vulkan-oriented redesign of ideas explored by DwarfStar, retargeted at Qwen3.6-35B-A3B and the AMD BC-250.

This work also depends heavily on the formats, quantization methods, kernels, tests and engineering knowledge developed by:

• llama.cpp
• GGML
• The Qwen team
• Salvatore Sanfilippo
• Georgi Gerganov
• llama.cpp and GGML contributors

License

The Qwen3.6-35B-A3B base model and these derived GGUF weights are distributed under the Apache License 2.0.

The q36 / QuarkStar inference-engine source code is distributed separately under the MIT License.

See the original model repository for the complete base-model license and usage terms:

Qwen/Qwen3.6-35B-A3B