---

license: other

base_model: Qwen/Qwen2.5-32B-Instruct

tags:

• llama-cpp
• gguf
• qwen
• mtp
• flashhead
• rocm
• speculative-decoding

pipeline_tag: text-generation

---

Qwen 3.6 35B-A3B UD Q4_K_M Selective Q4_0 FlashHead GGUF

High-performance GGUF artifact optimized for local high-throughput inference with FlashHead MTP speculative decoding on ROCm / HIP (e.g. AMD Strix Halo / gfx1151) and general llama.cpp deployments.

---

🚨 Required: Customized llama.cpp Fork

> [!IMPORTANT]

> This model requires a customized llama.cpp build.

> Standard upstream llama.cpp does not include the FlashHead draft head kernels or the multimodal slot save/restore patches required to operate this model with full acceleration.

>

> 🔗 Repository: https://github.com/AyaSakura-comp/llama.cpp/tree/flashhead-lm-head

> 🌿 Branch: flashhead-lm-head

Key Enhancements in the Custom Fork:

1. FlashHead Speculative Decoding Engine:

- Integrates custom Top-K candidate draft head for MTP (Multi-Token Prediction).

- Delivers 70% ~ 95%+ draft acceptance rates on dense target layers with minimal APU/GPU compute overhead.

- Configurable via LLAMA_FLASHHEAD_PROBES environment variable (e.g. 256).

1. Multimodal Slot Snapshotting & KV Cache Persistence:

- Upstream llama.cpp intentionally disables /slots/:id?action=save|restore when --mmproj is attached.

- This fork patches slot lifecycle handlers to allow persistent KV cache snapshotting on disk even when multimodal projectors are active.

1. Qwen preserve_thinking Chat Template Patch:

- Fixes spurious empty <think>\n\n</think>\n\n injection when reasoning is disabled or empty in subsequent turns.

1. Hybrid / Recurrent Architecture Slot Alignment:

- Accurately aligns checkpoint pos_max with llama_memory_seq_pos_max during slot eviction and restore, preventing full re-prefill penalties.

---

🛠️ How to Build the Customized llama.cpp

1. Clone the Fork

git clone -b flashhead-lm-head https://github.com/AyaSakura-comp/llama.cpp.git
cd llama.cpp

2. Build for Your Hardware

AMD ROCm / HIP (e.g. AMD Strix Halo / Radeon gfx1151 APU)

cmake -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j$(nproc) --target llama-server

NVIDIA CUDA

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j$(nproc) --target llama-server

CPU (AVX-512 / ARM Neon)

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc) --target llama-server

---

🚀 Recommended Launch Configuration

Run llama-server with FlashHead MTP speculative decoding enabled:

# Optional: Set FlashHead probe budget (default: 256)
export LLAMA_FLASHHEAD_PROBES=256

./build/bin/llama-server \
  -m Qwen3.6-35B-A3B-UD-Q4_K_M-selective-Q4_0-flashhead.gguf \
  --mmproj mmproj.gguf \
  --port 8001 \
  --host 0.0.0.0 \
  -ngl 99 \
  -fit off \
  -fa 1 \
  -c 260000 \
  -b 4096 \
  -ub 2048 \
  --alias qwen3.6-35b-q4 \
  --spec-type mtp \
  --spec-draft-n-max 3 \
  --slot-save-path ~/.cache/llama-slots/

Key Parameters Explained:

• --spec-type mtp --spec-draft-n-max 3: Activates multi-token speculative draft decoding (3 tokens per step).
• --mmproj mmproj.gguf: Enables multimodal vision capabilities.
• -c 260000 -b 4096 -ub 2048: Configures context size and batching parameters for high throughput.
• --slot-save-path ~/.cache/llama-slots/: Directory for fast slot snapshotting and instant session restores.

---

⚡ Companion Tool: pi-kv-cache-manager

If you are using this model with coding agents such as Pi Coding Agent, check out:

🔗 AyaSakura-comp/pi-kv-cache-manager

• Manages NVMe slot snapshots, LRU eviction, and system prompt prefix caching.
• Cuts long-context cold prefill latency from ~170s to 0.3s (543x speedup) when resuming or switching long-context sessions.

---

📦 Model Specifications

| Parameter | Value |

| :--- | :--- |

| Quantization | UD Q4_K_M + Selective Q4_0 MoE Down-weighting |

| Speculative Decoding | Embedded FlashHead Draft Tables |

| Primary Model File | Qwen3.6-35B-A3B-UD-Q4_K_M-selective-Q4_0-flashhead.gguf (~21.1 GB) |

| Multimodal Projector | mmproj.gguf (~861 MB) |

| Context Window | Up to 260,000+ tokens |