---

license: apache-2.0

base_model: Qwen/Qwen3.6-35B-A3B

tags:

• gguf
• llama.cpp
• mtp
• imatrix
• reasoning
• distillation
• moe

pipeline_tag: text-generation

---

Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — GGUF (native MTP + imatrix)

Lynn Agent high-end local option

This GGUF repo is the high-end local model option for Lynn Agent. Lynn v0.85.6+ recommends the 27B Q5_K_M GGUF as the default local path and exposes this 35B-A3B Q5_K_M GGUF for 32GB+ VRAM / unified-memory machines.

• Download Lynn Agent: GitHub Releases v0.85.6
• High-end quant in Lynn: Q5_K_M imatrix + native MTP (--spec-draft-n-max 3)
• Default local path: 27B Q5_K_M GGUF for 24GB+ machines; 9B / 4B remain manual low-config downgrades.

GGUF quantizations of nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill — a DeepSeek-V4-Pro ReAct-style reasoning distillation of Qwen3.6-35B-A3B (MoE, 35B total / ~3B active). Every GGUF carries a native MTP head and is imatrix-corrected (except Q8_0/F16). The MoE architecture plus MTP speculative decoding gives an especially strong single-stream speedup (below).

Quantizations

| File | Size | Notes |

|---|---|---|

| …-MTP-Q4_K_M-imatrix.gguf | ~21 GB | recommended balance |

| …-MTP-Q5_K_M-imatrix.gguf | ~25 GB | higher quality |

| …-MTP-Q6_K-imatrix.gguf | ~29 GB | near-lossless |

| …-MTP-Q8_0.gguf | ~37 GB | virtually lossless |

| …-MTP-F16.gguf | ~69 GB | full-precision reference |

| …-mmproj-F16.gguf | ~1 GB | vision projector (image input) |

Evaluation — distill vs base, Q4_K_M GGUF, same harness, thinking-on

| Metric | Distill | Base | Δ |

|---|---|---|---|

| GPQA-Diamond-198 | 80.3% | 72.7% | +7.6pp |

| MMLU-500 (5-shot) | 90.2% | 91.4% | −1.2 |

| End-to-end orchestration | 26.6s | 60.7s | 2.3× faster |

| Orchestration false-verify (20×5) | 0/20 | 0/20 | |

| GPQA non-terminating empties | 1 | 12 | −11 |

GPQA jumps +7.6pp and orchestration is 2.3× faster — the distill stops over-thinking and converges (non-terminating empties 12→1). MMLU dips 1.2pp: the 35B-A3B has less spare capacity than the 27B dense, so distillation slightly crowds out knowledge (a known trade-off).

MTP — native multi-token-prediction (MoE → strong speedup)

The GGUF has a native MTP head (mainline llama.cpp --spec-type draft-mtp). MoE benefits more from speculative decoding than dense. Single-stream measured on DGX Spark GB10:

| Quant | base t/s | MTP t/s | speedup |

|---|---|---|---|

| Q4_K_M | 79.2 | 86.3 | 1.09× |

| Q5_K_M | 73.0 | 88.0 | 1.20× |

| Q6_K | 66.0 | 75.2 | 1.14× |

| Q8_0 | 49.8 | 81.2 | 1.63× |

Higher quant tiers gain more (the bandwidth-bound base is accelerated by a cheap draft). Lossless (greedy speculative). Concurrency degrades speculation — use bare multi-concurrency for throughput.

Run

llama-server -m Qwen3.6-35B-A3B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

Thinking model — use temp 0.6 / top_p 0.95. For image input add --mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf.

---

中文

nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 的 GGUF 量化版 —— Qwen3.6-35B-A3B(MoE,35B 总参 / 约 3B 激活)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了原生 MTP 头,并做了 imatrix 校正(Q8_0/F16 除外)。MoE 架构 + MTP 投机解码,单流加速尤其明显(见下)。

量化档位

| 文件 | 大小 | 说明 |

|---|---|---|

| …-MTP-Q4_K_M-imatrix.gguf | ~21 GB | 推荐平衡档 |

| …-MTP-Q5_K_M-imatrix.gguf | ~25 GB | 更高质量 |

| …-MTP-Q6_K-imatrix.gguf | ~29 GB | 接近无损 |

| …-MTP-Q8_0.gguf | ~37 GB | 几乎无损 |

| …-MTP-F16.gguf | ~69 GB | 全精度参考 |

| …-mmproj-F16.gguf | ~1 GB | 视觉投影(图像输入) |

评测 —— 蒸馏 vs 原版,Q4_K_M GGUF,同 harness,thinking-on

| 维度 | 蒸馏 | 原版 | Δ |

|---|---|---|---|

| GPQA-Diamond-198 | 80.3% | 72.7% | +7.6pp |

| MMLU-500(5-shot)| 90.2% | 91.4% | −1.2 |

| 端到端编排 | 26.6s | 60.7s | 快 2.3× |

| 编排假验证(20×5)| 0/20 | 0/20 | |

| GPQA 未收口空答 | 1 | 12 | −11 |

GPQA +7.6pp、编排快 2.3×(学会收口,空答 12→1)。MMLU 微降 1.2:35B-A3B 容量比 27B dense 紧,蒸馏轻微挤占知识(已知权衡)。

MTP —— 原生多 token 预测(MoE → 加速更猛)

GGUF 内含原生 MTP 头(mainline llama.cpp --spec-type draft-mtp)。MoE 比 dense 更吃投机解码红利。DGX Spark GB10 单流实测:

| 量化档 | 裸 t/s | MTP t/s | 加速 |

|---|---|---|---|

| Q4_K_M | 79.2 | 86.3 | 1.09× |

| Q5_K_M | 73.0 | 88.0 | 1.20× |

| Q6_K | 66.0 | 75.2 | 1.14× |

| Q8_0 | 49.8 | 81.2 | 1.63× |

越高档加速越明显(带宽受限的裸推理被廉价草稿加速)。无损(贪心投机)。并发会摊薄投机 —— 高吞吐用裸多并发。

运行

llama-server -m Qwen3.6-35B-A3B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

思考型模型 —— 必须 temp 0.6 / top_p 0.95。图像输入加 --mmproj Qwen3.6-35B-A3B-DSV4Pro-Distill-mmproj-F16.gguf。

Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic /v1/messages 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。GGUF 走 LM Studio：LM Studio 0.4.1+ 加载本仓 GGUF（Q5_K_M 推荐），自带 Claude Code /v1/messages 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 FP8 仓 或 BF16 主仓，加 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3，模型名用 --served-model-name（不带斜杠）。参考 LM Studio · vLLM Claude Code。

---

Claude Code (experimental)

Claude Code needs an Anthropic-compatible /v1/messages backend + reliable tool calling — you cannot hand it a GGUF path. GGUF goes via LM Studio: LM Studio 0.4.1+ loads this repo's GGUF (Q5_K_M recommended) and exposes a built-in Claude Code /v1/messages endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the FP8 repo or the BF16 parent with --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3, model name = --served-model-name (no slashes).