---

license: apache-2.0

base_model: Qwen/Qwen3.6-35B-A3B

tags:

- gguf

- rotorquant

- kv-cache-quantization

- qwen

- qwen-3.6

- qwen3.6

- moe

- multimodal

- instruct

- llama-cpp

- quantized

library_name: gguf

pipeline_tag: image-text-to-text

---

> [!TIP]

> KV-cache quantization without any fork (recommended, 2026): upstream

> llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0

> (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or

> -ctk q4_0 -ctv q4_0 (~quarter memory, ≈7.6% perplexity increase). In

> Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep

> K and V types symmetric to stay on the fast fused Flash-Attention path.

> Since April 2026, mainline llama.cpp also applies Hadamard rotation to

> KV activations (PR #21038),

> which greatly improves low-bit KV quality (opt-out:

> LLAMA_ATTN_ROT_DISABLE=1).

>

> The RotorQuant/TurboQuant fork flow below is experimental/legacy: the

> TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork

> is unmaintained relative to mainline. It is NOT required to use this model.

<!-- kv-upstream-note -->

Qwen3.6-35B-A3B-RotorQuant-GGUF-Q2_K

GGUF Q2_K weight-quantized variant of Qwen/Qwen3.6-35B-A3B optimised for use with RotorQuant KV cache compression via a dedicated llama.cpp fork.

> Important: RotorQuant KV cache types (planar3, iso3) are not available in upstream llama.cpp, standard Ollama, or LM Studio.

> They require a specific llama.cpp fork.

> The GGUF file itself is a standard GGUF and works with any llama.cpp-compatible runtime using normal KV cache types (f16, q8_0, q4_0, etc.).

Hardware compatibility

| Device | VRAM / RAM | Recommendation |

| --- | --- | --- |

| CPU host with ≥14 GB RAM | ~14.0 GB | works via llama.cpp; slower than GPU but no accelerator required |

| Apple Silicon (Metal) | ~15.2 GB | llama.cpp Metal backend; fast on M-series unified memory |

| NVIDIA GPU (partial offload) | split between GPU + RAM | offload as many layers as VRAM allows; rest on CPU |

Overview

This model combines two independent compression techniques:

| Technique | What it does | Requirement |

|-----------|-------------|-------------|

| GGUF Q2_K weight quantization | Reduces model size from ~70 GB (BF16) to ~12.7 GB | Any llama.cpp-compatible runtime |

| RotorQuant KV cache compression — block-diagonal Clifford-algebra rotors for 3-bit KV cache (--cache-type-k iso3 --cache-type-v iso3) | Block-diagonal rotations / random rotation for compressed KV cache | llama-cpp-turboquant fork only |

Quickstart

Option A — RotorQuant KV cache (experimental fork — not required)

You must build from the RotorQuant-enabled llama.cpp fork:

# Clone and build the fork
git clone https://github.com/johndpope/llama-cpp-turboquant.git
cd llama-cpp-turboquant && git checkout feature/planarquant-kv-cache

# CUDA (Windows/Linux)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# Metal (Apple Silicon)
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# Run with RotorQuant KV cache
./build/bin/llama-cli -m Qwen3.6-35B-A3B-RotorQuant-GGUF-Q2_K.gguf \
  --cache-type-k iso3 --cache-type-v iso3 \
  -ngl 99 -fa \
  -p "Explain quantum computing"

# Or run as a server
./build/bin/llama-server -m Qwen3.6-35B-A3B-RotorQuant-GGUF-Q2_K.gguf \
  --cache-type-k iso3 --cache-type-v iso3 \
  -ngl 99 -fa --jinja

Option B — With standard llama.cpp / LM Studio / Ollama

The GGUF works as a normal quantised model. You won't get RotorQuant-specific KV cache benefits, but standard KV cache quantization (q8_0, q4_0) still reduces VRAM significantly.

llama.cpp (upstream)

llama-cli -m Qwen3.6-35B-A3B-RotorQuant-GGUF-Q2_K.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99 -fa \
  -p "Explain quantum computing"

LM Studio

1. Download the GGUF file and load in LM Studio.
2. Enable Developer Mode (Settings → Developer).
3. In the model loader's advanced settings, set Flash Attention to ON.
4. Set K Cache Quantization and V Cache Quantization to q8_0 (or q4_0 for more aggressive VRAM savings).
5. Note: LM Studio does not currently support RotorQuant's iso3 cache types. Track this feature request for updates.

Ollama

# Standard Ollama does not support RotorQuant cache types.
# Use with default or q8_0 KV cache via OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_FLASH_ATTENTION=1 ollama run majentik/Qwen3.6-35B-A3B-RotorQuant-GGUF-Q2_K

Specifications

| Property | Value |

|----------|-------|

| Base Model | Qwen/Qwen3.6-35B-A3B |

| Architecture | Sparse MoE (3B active per token), instruct-tuned |

| Parameters | 35B total, 3B active per token (MoE) |

| Context Length | 262K native |

| Weight Quantization | GGUF Q2_K (aggressive 2-bit, noticeable quality drop) |

| Original Size (BF16) | ~70 GB (approx.) |

| Quantized File Size | ~12.7 GB |

| KV Cache (RotorQuant) | 3-bit via --cache-type-k iso3 --cache-type-v iso3 (fork only) |

| KV Cache (standard) | q8_0, q4_0, f16, etc. (any llama.cpp runtime) |

| License | apache-2.0 |

| Modalities | Text + Image + Video (multimodal) |

| Compatible Runtimes | llama.cpp, LM Studio, Ollama, koboldcpp |

About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's release labels, not distinct

quantization algorithms — for any given tier, both brand repos carry

byte-identical weights produced with the standard MLX / llama.cpp quantizers.

No brand-specific speedup is claimed or measured. The KV-cache fork these

labels originally referred to is legacy; for KV-cache memory savings use the

upstream options described above (-ctk/-ctv q8_0, OLLAMA_KV_CACHE_TYPE).

Current Status of RotorQuant in the Ecosystem

| Runtime | RotorQuant Support | Standard KV Quant |

|---------|---------------------|-------------------|

| llama.cpp (upstream) | ❌ Not merged | ✅ q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 |

| llama-cpp-turboquant fork | ✅ planar3, iso3 | ✅ All standard types |

| LM Studio | ❌ Requested | ✅ Via advanced settings |

| Ollama | ❌ Not supported | ✅ Via OLLAMA_KV_CACHE_TYPE |

| koboldcpp | ❌ Not supported | ✅ Standard types |

Recommended Settings

For VRAM-constrained setups, standard q8_0 KV cache quantization already halves KV cache memory with negligible quality impact. Flash Attention should always be enabled — it is required for V cache quantization and improves memory efficiency regardless.

| VRAM | Suggested Configuration |

|------|------------------------|

| 24 GB (RTX 4090) | Q2_K + q8_0 KV cache + Flash Attention, 8K–16K context |

| 16 GB | Q2_K + q4_0 KV cache + Flash Attention, 4K–8K context |

| 48+ GB | Q2_K + f16 KV cache, full 32K+ context |

See Also

• RotorQuant GitHub
• llama-cpp-turboquant fork
• TurboQuant llama.cpp discussion
• TurboQuant paper (arXiv: 2504.19874)
• Base model: Qwen/Qwen3.6-35B-A3B
• Qwen3.6-35B-A3B on HuggingFace

Quant trade-off (GGUF lane)

| Quant | Approx size | Use case | Recommendation |

|---|---|---|---|

| Q2_K | ~19 GB | Lossy, low-RAM CPU/edge | Resource-constrained inference |

| Q3_K_M | ~21 GB | Smaller-than-Q4, modest quality drop | Edge devices with ~16 GB RAM |

| IQ4_XS | ~18 GB | Importance-quant 4-bit, smaller than Q4_K_M | Best size/quality at 4-bit |

| Q4_K_M | ~26 GB | Balanced default | Recommended for most users |

| Q5_K_M | ~27 GB | Higher fidelity than Q4 | Quality-sensitive applications |

| Q6_K | ~32 GB | Approaching FP16 quality | High-fidelity CPU/edge |

| Q8_0 | ~36 GB | Near-lossless reference | Fidelity-critical work |

| MXFP4_MOE | ~19 GB | Microscaling FP4 (MoE-aware) | vLLM / transformers users |

(Current variant — Q2_K — is bolded.)

Variants in this family

(Showing 24 sibling variants under majentik/qwen3.6-35b-a3b-*. The current variant — RotorQuant-GGUF-Q2_K — is bolded.)

| Variant | Runtime | Approx size | Use case |

|---|---|---|---|

| RotorQuant-GGUF-IQ4_XS | llama.cpp | ~30 GB | Lossy 4-bit, low-RAM CPU/edge |

| RotorQuant-GGUF-Q2_K | llama.cpp | ~21 GB | Lossy, low-RAM CPU/edge |

| RotorQuant-GGUF-Q3_K_M | llama.cpp | ~27 GB | Smaller 3-bit, CPU-friendly |

| RotorQuant-GGUF-Q4_K_M | llama.cpp | ~38 GB | Balanced default |

| RotorQuant-GGUF-Q5_K_M | llama.cpp | ~46 GB | Higher fidelity, more RAM |

| RotorQuant-GGUF-Q8_0 | llama.cpp | ~74 GB | Near-lossless reference |

| RotorQuant-MLX-2bit | mlx-lm | ~11 GB | Apple Silicon, smallest |

| RotorQuant-MLX-3bit | mlx-lm | ~16 GB | Apple Silicon, small |

| RotorQuant-MLX-4bit | mlx-lm | ~22 GB | Apple Silicon balanced |

| RotorQuant-MLX-5bit | mlx-lm | ~27 GB | Apple Silicon, higher fidelity |

| RotorQuant-MLX-6bit | mlx-lm | ~32 GB | Apple Silicon, near-lossless |

| RotorQuant-MLX-8bit | mlx-lm | ~41 GB | Apple Silicon reference |

| TurboQuant-MLX-2bit | mlx-lm | ~11 GB | Apple Silicon, smallest |

| TurboQuant-MLX-3bit | mlx-lm | ~16 GB | Apple Silicon, small |

| TurboQuant-MLX-4bit | mlx-lm | ~22 GB | Apple Silicon balanced |

| TurboQuant-MLX-5bit | mlx-lm | ~27 GB | Apple Silicon, higher fidelity |

| TurboQuant-MLX-6bit | mlx-lm | ~32 GB | Apple Silicon, near-lossless |

| TurboQuant-MLX-8bit | mlx-lm | ~41 GB | Apple Silicon reference |