---

license: apache-2.0

base_model: Qwen/Qwen3.6-27B

tags:

• gguf
• llama.cpp
• mtp
• imatrix
• reasoning
• distillation

pipeline_tag: text-generation

---

Qwen3.6-27B-DSV4Pro-Thinking-Distill — GGUF (native MTP + imatrix)

Lynn Agent default local model

This GGUF repo is the default local model recommendation for Lynn Agent. Lynn v0.85.6+ uses Qwen3.6-27B-DSV4Pro-Distill-MTP-Q5_K_M-imatrix.gguf as the first-choice edge model for 24GB+ VRAM / unified-memory machines; 32GB+ machines can choose the 35B-A3B Q5_K_M path, while 9B / 4B remain explicit low-config downgrades.

• Download Lynn Agent: GitHub Releases (current desktop/CLI release: v0.85.6)
• Default quant in Lynn: Q5_K_M imatrix + native MTP (--spec-draft-n-max 3)
• Recommended use: local agentic reasoning, coding, and cloud-independent fallback inside Lynn.

GGUF quantizations of nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill — a DeepSeek-V4-Pro ReAct-style reasoning distillation of Qwen3.6-27B (dense). Every GGUF carries a native MTP (multi-token-prediction) head and is imatrix-corrected (except Q8_0/BF16). Distillation only changes the thinking style (the model learns to converge), not coding or knowledge.

Quantizations

| File | Size | Notes |

|---|---|---|

| …-MTP-IQ4_XS-imatrix.gguf | 15.3 GB | 16 GB-VRAM floor — fits 16 GB no-offload; smallest we ship |

| …-MTP-Q4_K_S-imatrix.gguf | 15.8 GB | speed-first |

| …-MTP-Q4_K_M-imatrix.gguf | 16.8 GB | recommended balance |

| …-MTP-Q5_K_M-imatrix.gguf | 19.5 GB | higher quality |

| …-MTP-Q6_K-imatrix.gguf | 22.4 GB | near-lossless |

| …-MTP-Q8_0.gguf | 29.0 GB | virtually lossless |

| …-MTP-BF16.gguf | 54.7 GB | full-precision reference |

| …-mmproj-F16.gguf | 0.9 GB | vision projector (image input) |

> IQ4_XS is the floor. Sized for a 16 GB-VRAM card with zero CPU offload (14.3 GiB weights + headroom for KV). Smoke-verified converging (3/3 reasoning prompts correct, 0 length-runaway, EN + ZH) at 31 t/s single-stream with the native MTP head (--spec-type draft-mtp, accept 0.80, mean accept-len 3.4). This is the smallest tier we ship — below IQ4_XS a 27B reasoning distill starts to lose convergence, so we stop here.

> 📏 GB vs GiB — why a 15.3 GB file fits a "16 GB" GPU. Sizes in this table are decimal GB (10⁹ bytes). GPU VRAM is built in GiB (2³⁰ bytes) but marketed as "GB" — so a "16 GB" card is really 16 GiB ≈ 17.2 GB. IQ4_XS (15.3 GB = 14.3 GiB) therefore loads fully on a 16 GiB card with ~1.7 GiB left for KV cache — no CPU offload. Rule of thumb: a file runs no-offload when its GiB size < the card's nominal "GB" number (here 14.3 < 16 ✓).

Evaluation — distill vs base, same-spec GGUF, thinking-on

Both rows use the same quant + MTP; the only variable is "distilled or not". Do not cross-compare quant tiers (Q4 and Q5 use different harnesses).

Q4_K_M (archived harness):

| Dimension | Distill | Base | Δ |

|---|---|---|---|

| GPQA-Diamond-198 | 80.81% (160/198) | 73.7% | +7.1 |

| MMLU-500 (5-shot) | 91.8% | 91.6% | +0.2 |

| GPQA unconverged empties | 0 | 14 | −14 |

| coding-100 (real sandbox) | 86/100 | 83/100 | +3 |

Q5_K_M (streaming harness · conc4 · timeout 1800 · max_tokens 32000, finish_reason logged):

| Dimension | Distill | Base | Δ |

|---|---|---|---|

| GPQA-Diamond-198 | 81.82% (162/198) | 68.69% (136/198) | +13.13pp |

| MMLU-500 | 90.0% | 89.6% | +0.4 |

| GPQA finish=stop (converged) | 198/198 | 186/198 | |

| GPQA finish=length (hit 32K wall) | 0 | 12 | |

Hard reasoning jumps (GPQA +7~13pp), knowledge holds, and the distill learns to converge — the base hits the 32K wall on 12 questions (never answers); the distill converges on all 198.

MTP — native multi-token-prediction (single-stream speedup)

The GGUF has a native MTP head (mainline llama.cpp --spec-type draft-mtp, no external draft model). Measured single-stream on DGX Spark GB10, Q4_K_M:

| --spec-draft-n-max | TPS | accept rate | accept len |

|---|---|---|---|

| bare (no MTP) | 10.4 | — | — |

| n-max=3 ⭐ | 26.8 | 0.72 | 3.16 |

2.3–2.6× single-stream speedup, lossless (greedy speculative only accepts the target-argmax token). Speculation is single-stream — concurrency degrades it, so use bare multi-concurrency for throughput.

Run

llama-server -m Qwen3.6-27B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

Thinking model — use temp 0.6 / top_p 0.95 (greedy loops to death). For image input add --mmproj Qwen3.6-27B-DSV4Pro-Distill-mmproj-F16.gguf.

---

中文

nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill 的 GGUF 量化版 —— Qwen3.6-27B(dense)的 DeepSeek-V4-Pro ReAct 思维蒸馏版。每个档位都焊了原生 MTP 头,并做了 imatrix 校正(Q8_0/BF16 除外)。蒸馏只改思维方式(学会"收口"),coding 与知识不降。

量化档位

| 文件 | 大小 | 说明 |

|---|---|---|

| …-MTP-IQ4_XS-imatrix.gguf | 15.3 GB | 16G 显存底线 —— 16G 不 offload 可跑,最小档 |

| …-MTP-Q4_K_S-imatrix.gguf | 15.8 GB | 速度优先 |

| …-MTP-Q4_K_M-imatrix.gguf | 16.8 GB | 平衡档 / 低显存回退 |

| …-MTP-Q5_K_M-imatrix.gguf | 19.5 GB | Lynn 默认推荐(24GB+) |

| …-MTP-Q6_K-imatrix.gguf | 22.4 GB | 接近无损 |

| …-MTP-Q8_0.gguf | 29.0 GB | 几乎无损 |

| …-MTP-BF16.gguf | 54.7 GB | 全精度参考 |

| …-mmproj-F16.gguf | 0.9 GB | 视觉投影(图像输入) |

> IQ4_XS 是底线档。 为 16 GB 显存、不 offload 而做(14.3 GiB 权重 + KV 余量)。冒烟实测收口干净(3/3 推理题正确、0 长度跑飞、中英),单流 31 t/s + 原生 MTP 头(--spec-type draft-mtp,接受率 0.80、平均接受长 3.4)。这是我们发布的最小档 —— 27B 推理蒸馏低于 IQ4_XS 会开始丢收口,到此为止。

> 📏 GB vs GiB —— 为什么 15.3 GB 的文件能进"16 GB"显卡。 本表大小用十进制 GB(10⁹ 字节);显卡显存按 GiB(2³⁰ 字节) 造却标"GB",所以一张"16 GB"卡其实是 16 GiB ≈ 17.2 GB。于是 IQ4_XS(15.3 GB = 14.3 GiB)能完整载入 16 GiB 卡,剩 ~1.7 GiB 给 KV —— 不用 offload。经验法则:文件的 GiB 数 < 显卡标称"GB"数,就能不 offload 跑(这里 14.3 < 16 ✓)。

评测 —— 蒸馏 vs 原版,同规格 GGUF,thinking-on

两行同量化 + 同 MTP,唯一变量是"是否蒸馏"。禁止跨量化档比较(Q4 与 Q5 用了不同 harness)。

Q4_K_M(旧 harness):

| 维度 | 蒸馏 | 原版 | Δ |

|---|---|---|---|

| GPQA-Diamond-198 | 80.81%(160/198)| 73.7% | +7.1 |

| MMLU-500(5-shot)| 91.8% | 91.6% | +0.2 |

| GPQA 未收口空答 | 0 | 14 | −14 |

| coding-100(真实沙箱)| 86/100 | 83/100 | +3 |

Q5_K_M(流式 harness · 并发4 · timeout 1800 · max_tokens 32000,逐题记 finish_reason):

| 维度 | 蒸馏 | 原版 | Δ |

|---|---|---|---|

| GPQA-Diamond-198 | 81.82%(162/198)| 68.69%(136/198)| +13.13pp |

| MMLU-500 | 90.0% | 89.6% | +0.4 |

| GPQA finish=stop(收口)| 198/198 | 186/198 | |

| GPQA finish=length(撞 32K 墙)| 0 | 12 | |

硬推理大涨(GPQA +7~13pp),知识不降,且蒸馏学会收口 —— 原版有 12 题撞 32K 墙(始终不出答案),蒸馏 198 题全收口。

MTP —— 原生多 token 预测(单流加速)

GGUF 内含原生 MTP 头(mainline llama.cpp --spec-type draft-mtp,无需外挂草稿模型)。DGX Spark GB10 单流实测,Q4_K_M:

| --spec-draft-n-max | TPS | 接受率 | 接受长度 |

|---|---|---|---|

| 裸(无 MTP)| 10.4 | — | — |

| n-max=3 ⭐ | 26.8 | 0.72 | 3.16 |

单流 2.3–2.6× 加速,无损(贪心投机只接受目标 argmax token)。投机是单流手段 —— 并发会摊薄它,高吞吐场景用裸多并发。

运行

llama-server -m Qwen3.6-27B-DSV4Pro-Distill-MTP-Q4_K_M-imatrix.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 8192 --jinja --host 127.0.0.1 --port 8080

思考型模型 —— 必须 temp 0.6 / top_p 0.95(贪心会死循环)。图像输入加 --mmproj Qwen3.6-27B-DSV4Pro-Distill-mmproj-F16.gguf。

Claude Code（实验性 / experimental）

Claude Code 需要兼容 Anthropic /v1/messages 的后端 + 可靠的工具调用，不能直接喂 GGUF 文件路径。GGUF 走 LM Studio：LM Studio 0.4.1+ 加载本仓 GGUF（Q5_K_M 推荐），自带 Claude Code /v1/messages 端点，Claude Code 指过去即可。

vLLM 路线需 safetensors —— 用 FP8 仓 或 BF16 主仓，加 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3，模型名用 --served-model-name（不带斜杠）。参考 LM Studio · vLLM Claude Code。

---

Claude Code (experimental)

Claude Code needs an Anthropic-compatible /v1/messages backend + reliable tool calling — you cannot hand it a GGUF path. GGUF goes via LM Studio: LM Studio 0.4.1+ loads this repo's GGUF (Q5_K_M recommended) and exposes a built-in Claude Code /v1/messages endpoint — just point Claude Code at it. The vLLM route needs safetensors — use the FP8 repo or the BF16 parent with --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3, model name = --served-model-name (no slashes).