SC117/QwenPaw-Flash-9B-MTP-GGUF overview

---

language:

• en
• zh

base_model:

• Qwen/Qwen3.5-9B
• agentscope-ai/QwenPaw-Flash-9B

tags:

• mtp
• speculative-decoding
• qwen3.5
• gguf
• agent
• tool-call

license: apache-2.0

---

<div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; display: flex; flex-direction: column; gap: 20px; margin-bottom: 30px;">

<div style="border: 1px solid #cbd5e1; border-radius: 16px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 8px rgba(0,0,0,0.04);">

<div style="background: linear-gradient(135deg, #007aff 0%, #5856d6 100%); padding: 28px 24px; color: white; text-align: center; position: relative; overflow: hidden;">

<div style="position: absolute; top: -30px; right: -30px; width: 120px; height: 120px; background: rgba(255,255,255,0.1); border-radius: 50%;"></div>

<div style="position: absolute; bottom: -20px; left: 40px; width: 80px; height: 80px; background: rgba(255,255,255,0.08); border-radius: 50%;"></div>

<h1 style="margin: 0; font-size: 24px; font-weight: 700; position: relative; z-index: 1;">⚡ QwenPaw-Flash-9B-MTP</h1>

<p style="margin: 8px 0 0 0; font-size: 15px; opacity: 0.95; position: relative; z-index: 1;">MTP Speculative Decoding · Agent-Optimized · 1.5–4.1× Speedup</p>

<p style="margin: 8px 0 0 0; font-size: 14px; position: relative; z-index: 1;"><a href="https://huggingface.co/SC117/QwenPaw-Flash-9B-MTP-GGUF/blob/main/README_zh.md" style="color: #ffffff; text-decoration: none;">📖 中文文档</a></p>

</div>

<div style="padding: 16px 24px; background: #f8fafc; border-bottom: 1px solid #e2e8f0; display: flex; flex-wrap: wrap; gap: 8px;">

<span style="background: #dbeafe; color: #1e40af; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">MTP</span>

<span style="background: #d1fae5; color: #065f46; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">Dense 9B</span>

<span style="background: #fef3c7; color: #92400e; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">Q6_K / Q8_0 / Q4_K_M</span>

<span style="background: #ede9fe; color: #5b21b6; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">262K Context</span>

<span style="background: #fce7f3; color: #9d174d; padding: 4px 12px; border-radius: 20px; font-size: 12px; font-weight: 600;">Vision Encoder</span>

</div>

<div style="padding: 20px 24px;">

<p style="margin: 0; font-size: 14px; color: #334155; line-height: 1.7;">

The <b>original QwenPaw-Flash-9B</b> (no abliteration) with <b>MTP (Multi-Token Prediction) head</b> weights injected from the official <a href="https://huggingface.co/Qwen/Qwen3.5-9B" style="color: #c2410c; text-decoration: none;">Qwen3.5-9B</a> base model. By reconstructing the MTP speculative decoding head — stripped during QwenPaw fine-tuning — this model achieves <b>up to 4.1× inference speedup</b> while maintaining accuracy. Heretic (uncensored) version: <a href="https://huggingface.co/SC117/QwenPaw-Flash-9B-heretic-MTP-GGUF" style="color: #c2410c; text-decoration: none;">QwenPaw-Flash-9B-heretic-MTP-GGUF</a>

</p>

</div>

</div>

<div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 4px rgba(0,0,0,0.02);">

<div style="background: linear-gradient(135deg, #007aff 0%, #5856d6 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px; display: flex; align-items: center; gap: 8px;"><span>⚡</span> MTP Speculative Decoding</div>

<div style="padding: 16px;">

<table style="width: 100%; border-collapse: collapse; font-family: inherit; font-size: 13px;">

<tbody>

<tr><td style="padding: 10px; color: #334155; line-height: 1.7;" colspan="2"><b>What is MTP?</b><br>Multi-Token Prediction (MTP) is a speculative decoding technique where a small "draft head" predicts multiple future tokens in parallel. The main model then verifies these drafts in a single forward pass, accepting correct predictions for up to 2–4× speedup in practice.</td></tr>

<tr><td style="padding: 10px; color: #334155; line-height: 1.7;" colspan="2"><b>Injection Method</b><br>The official Qwen3.5-9B ships with a 4-layer MTP head (~243M params). During QwenPaw fine-tuning, the MTP head weights were stripped — only the config placeholder <code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">mtp_num_hidden_layers: 1</code> remained.</td></tr>

<tr><td style="padding: 10px; color: #334155; line-height: 1.7;" colspan="2"><b>Recovery Process</b><br>1. Download official <a href="https://huggingface.co/Qwen/Qwen3.5-9B" style="color: #c2410c; text-decoration: none;">Qwen3.5-9B</a> base model weights<br>2. Extract MTP layer weights (15 tensors starting with <code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">mtp.</code>)<br>3. Merge into <a href="https://huggingface.co/agentscope-ai/QwenPaw-Flash-9B" style="color: #c2410c; text-decoration: none;">QwenPaw-Flash-9B</a> safetensors<br>4. Convert to GGUF with <code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">convert_hf_to_gguf.py</code> (with MTP support)<br>5. Quantize with <code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">llama-quantize</code></td></tr>

<tr><td style="padding: 10px; color: #334155; line-height: 1.7;" colspan="2"><b>Why This Works</b><br>The MTP head is a lightweight 4-layer MLP decoder that maps the main model's last hidden state to future token logits. It sits entirely in <b>speculative decoding space</b> — the main model's weights are <b>unchanged</b>, so no fine-tuning is needed. The head simply needs to exist with compatible dimensions for llama.cpp's <code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">--spec-type draft-mtp</code> to activate.</td></tr>

</tbody>

</table>

</div>

</div>

<div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 4px rgba(0,0,0,0.02);">

<div style="background: linear-gradient(135deg, #059669 0%, #10b981 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px; display: flex; align-items: center; gap: 8px;"><span>🏗️</span> Architecture</div>

<div style="padding: 16px;">

<table style="width: 100%; border-collapse: collapse; font-family: inherit; font-size: 13px;">

<tbody>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600; width: 40%;">Type</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Qwen3_5ForConditionalGeneration (multimodal with vision encoder) + MTP spec head</td></tr>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Main Model</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">~9B parameters</td></tr>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">MTP Head</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">~243M parameters (2.7% overhead)</td></tr>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Layers</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">32 (hybrid: Gated DeltaNet + Gated Attention) + 4 MTP decoder layers</td></tr>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Context Length</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">262,144 tokens</td></tr>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #047857; font-weight: 600;">Speculative Decoding</td><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;"><code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">--spec-type draft-mtp</code> with <code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">--spec-draft-n-max 2</code></td></tr>

<tr><td style="padding: 8px 10px; color: #047857; font-weight: 600;">MTP Acceptance Rate</td><td style="padding: 8px 10px; color: #334155;">~50% (measured on heretic version)</td></tr>

</tbody>

</table>

</div>

</div>

<div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 4px rgba(0,0,0,0.02);">

<div style="background: linear-gradient(135deg, #0891b2 0%, #06b6d4 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px; display: flex; align-items: center; gap: 8px;"><span>📦</span> GGUF Files</div>

<div style="padding: 16px;">

<table style="width: 100%; border-collapse: collapse; font-family: inherit; font-size: 13px;">

<thead>

<tr style="background: rgba(8,145,178,0.05);">

<th style="padding: 8px 10px; border-bottom: 2px solid #0891b2; text-align: left; color: #0e7490; font-weight: bold;">File</th>

<th style="padding: 8px 10px; border-bottom: 2px solid #0891b2; text-align: left; color: #0e7490; font-weight: bold;">Size</th>

<th style="padding: 8px 10px; border-bottom: 2px solid #0891b2; text-align: left; color: #0e7490; font-weight: bold;">Type</th>

<th style="padding: 8px 10px; border-bottom: 2px solid #0891b2; text-align: left; color: #0e7490; font-weight: bold;">Notes</th>

</tr>

</thead>

<tbody>

<tr><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;"><code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">QwenPaw-Flash-9B-MTP-BF16.gguf</code></td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">17.14 GB</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">BF16</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Full precision, reference quality</td></tr>

<tr><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;"><code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">QwenPaw-Flash-9B-MTP-Q8_0.gguf</code></td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">9.11 GB</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Q8_0</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">~8.5 bpw, near-lossless</td></tr>

<tr><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;"><code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">QwenPaw-Flash-9B-MTP-Q6_K.gguf</code></td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">7.04 GB</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Q6_K</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">✅ <b>Recommended, best value</b></td></tr>

<tr><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;"><code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">QwenPaw-Flash-9B-MTP-Q4_K_M.gguf</code></td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">5.38 GB</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Q4_K_M</td><td style="padding: 7px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;">Compact, best size/quality tradeoff</td></tr>

<tr><td style="padding: 7px 10px; color: #334155;"><code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">mmproj-QwenPaw-Flash-9B-heretic-BF16.gguf</code></td><td style="padding: 7px 10px; color: #334155;">0.86 GB</td><td style="padding: 7px 10px; color: #334155;">BF16</td><td style="padding: 7px 10px; color: #334155;">Vision encoder (multimodal)</td></tr>

</tbody>

</table>

</div>

</div>

<div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 4px rgba(0,0,0,0.02);">

<div style="background: linear-gradient(135deg, #4b5563 0%, #6b7280 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px; display: flex; align-items: center; gap: 8px;"><span>🚀</span> Usage</div>

<div style="padding: 16px;">

<table style="width: 100%; border-collapse: collapse; font-family: inherit; font-size: 13px; margin-bottom: 16px;">

<tbody>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;" colspan="2"><b>With MTP Speculative Decoding</b></td></tr>

<tr><td style="padding: 4px 10px 8px 10px; color: #334155;" colspan="2"><div style="background: #1e293b; border-radius: 8px; padding: 12px;"><pre style="margin: 0; font-family: 'SF Mono', 'Fira Code', monospace; font-size: 12px; color: #e2e8f0; line-height: 1.6; white-space: pre-wrap;">llama-server -m QwenPaw-Flash-9B-MTP-Q6_K.gguf \

-ngl 99 -fa on -c 8192 \

--spec-type draft-mtp --spec-draft-n-max 2 \

--host 0.0.0.0 --port 8088</pre></div></td></tr>

<tr><td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); color: #334155;" colspan="2"><b>Without MTP (fallback)</b></td></tr>

<tr><td style="padding: 4px 10px 8px 10px; color: #334155;" colspan="2"><div style="background: #1e293b; border-radius: 8px; padding: 12px;"><pre style="margin: 0; font-family: 'SF Mono', 'Fira Code', monospace; font-size: 12px; color: #e2e8f0; line-height: 1.6; white-space: pre-wrap;"># Just omit spec args — works as a normal GGUF

llama-server -m QwenPaw-Flash-9B-MTP-Q6_K.gguf \

-ngl 99 -fa on -c 8192 \

--host 0.0.0.0 --port 8088</pre></div></td></tr>

</tbody>

</table>

<p style="margin: 0; font-size: 12px; color: #64748b; line-height: 1.5;">Compatible with llama.cpp, LM Studio, Jan, koboldcpp, and other GGUF runtimes. For MTP, use <code style="background: #f1f5f9; padding: 2px 6px; border-radius: 4px; font-size: 11px;">--spec-type draft-mtp</code>. The MTP head is a <b>lossless copy</b> from Qwen3.5-9B — no training involved.</p>

</div>

</div>

<div style="border: 1px solid #cbd5e1; border-radius: 12px; overflow: hidden; background: #ffffff; box-shadow: 0 2px 4px rgba(0,0,0,0.02);">

<div style="background: linear-gradient(135deg, #7c3aed 0%, #a855f7 100%); padding: 12px 16px; color: white; font-weight: 700; font-size: 14px; display: flex; align-items: center; gap: 8px;"><span>🔗</span> Credits</div>

<div style="padding: 16px;">

<p style="margin: 0; font-size: 13px; color: #334155; line-height: 1.7;">

<b>Base Model:</b> <a href="https://huggingface.co/agentscope-ai/QwenPaw-Flash-9B" style="color: #c2410c; text-decoration: none;">agentscope-ai/QwenPaw-Flash-9B</a><br>

<b>MTP Head Source:</b> <a href="https://huggingface.co/Qwen/Qwen3.5-9B" style="color: #c2410c; text-decoration: none;">Qwen/Qwen3.5-9B</a><br>

<b>Quantization Tool:</b> llama.cpp · <a href="https://github.com/ggml-org/llama.cpp" style="color: #c2410c; text-decoration: none;">GitHub</a>

</p>

</div>

</div>

</div>