---

license: apache-2.0

library_name: llama.cpp

pipeline_tag: text-generation

base_model:

• incoai/Qwen3.8-27B-DFlash2

inference: false

tags:

• gguf
• dflash2
• speculative-decoding
• draft-model
• llama.cpp
• mixed-precision
• low-bit
• memory-efficient

---

Qwen3.8-27B-DFlash2 — Q2_K_S-MIX draft (half the reference size)

A mixed-precision 2–3-bit quantization of the DFlash 2 draft model for

Qwen/Qwen3.8-27B, built to be

50% the size of the reference Q4_K_M checkpoint while retaining

~98% of its throughput.

Q2_K_S-MIX is a mixed-precision quant that

compresses the large feed-forward blocks hard while keeping the small,

high-impact tensors (the path selector and feature projection) more precise, so

it lands at half the reference size with only a small acceptance loss.

Measured performance (vs the reference Q4_K_M)

All draft models served with llama-server (DFlash 2, PR #27342) against the

Qwen3.8-27B target on a 24 GB NVIDIA GPU, one fixed conversational prompt with medium reasoning effort,

temperature 1.0, concurrency 1, 5 replicate runs. n_max is the draft block

length (speculative tokens drafted per verification step).

| n_max | Metric | Q4_K_M (1,090 MiB) | Q3_K_M (874 MiB) | Q2_K (673 MiB) | Q2_K_S-MIX (535 MiB) | Ratio (vs Q4) |

| :-- | :--- | ---: | ---: | ---: | ---: | :-- |

| 3 | acceptance | 0.539 | 0.543 | 0.525 | 0.524 | 0.97 |

| 4 | acceptance | 0.466 | 0.459 | 0.441 | 0.435 | 0.93 |

| 5 | acceptance | 0.403 | 0.403 | 0.388 | 0.373 | 0.93 |

| 3 | draft len | 2.62 | 2.63 | 2.57 | 2.57 | 0.98 |

| 4 | draft len | 2.86 | 2.83 | 2.76 | 2.74 | 0.96 |

| 5 | draft len | 3.01 | 3.01 | 2.93 | 2.86 | 0.95 |

| 3 | tok/s | 95.2 | 96.6 | 94.9 | 96.2 | 1.01 |

| 4 | tok/s | 99.2 | 98.3 | 96.1 | 95.6 | 0.96 |

| 5 | tok/s | 104.2 | 104.8 | 102.4 | 101.9 | 0.98 |

| — | Size | 1,090 MiB (4.76 bpw) | 874 MiB (3.81 bpw) | 673 MiB (2.93 bpw) | 535 MiB (2.33 bpw) | 0.49 |

Q2_K_S-MIX is the smallest draft (535 MiB) and sits on the size–throughput

frontier: it posts the lowest acceptance of the four but converts it into

within-a-few-percent throughput at every n_max. Non-imatrix Q2_K quants were strictly

worse than Q2_K_S-MIX — larger, with lower acceptance and throughput — so they

are omitted here.

The 2.33-bit draft model accepts slightly fewer tokens per step than the 4.76-bit

reference (e.g. 0.524 vs 0.539 acceptance, 2.57 vs 2.62 draft len at

n_max=3), which shows up as a small throughput gap (within a few percent at

each n_max). Because DFlash 2 is lossless, this costs speed, not quality —

for the same prompt the output is accepted by the same target at the same

quality; the smaller drafter just needs marginally more verification steps.

If you have the VRAM to spare, I highly recommmend the

Q3_K_M imatrix quant as an option — it posts higher acceptance at every n_max at the the cost of just a bit of context size when memory-constrained, however vision is broken on that release until you run the patcher script in this repo to add the required metadata.

Throughput vs draft size, by block length n_max:

<div align="center">

<img src="assets/tg_vs_mib.png" alt="Throughput vs draft size for n_max = 3, 4, 5" width="100%">

</div>

How it was built (changes vs the reference)

Built clean from the upstream BF16 drafter (incoai/Qwen3.8-27B-DFlash2

GGUF) with llama-quantize on a build

with DFlash 2 support (PR #27342).

No dequant-from-quant: the source is the full-precision checkpoint. It is a

q2_k_s base with per-tensor --tensor-type overrides, and the low-bit I-quants

are quantized with a real activation-calibrated importance matrix (imatrix)

— captured from the draft's own decode activations — rather than a flat one, so

each super-block is weighted by the magnitudes it actually sees:

| Component | Tensors | Quant |

| :--- | :--- | :--- |

| Feed-forward (SwiGLU gate/up/down) | ~69% of params | iq2_xxs |

| Token-path selector (hidden / predecessor / successor) | ~7% | hidden q5_k; pred/succ iq2_s |

| Feature projection fc | 5120 × 25600 | q3_k |

| Two-tap dynamic-conv projections (attn + ffn) | 2 × 5 blocks | iq2_xxs |

| Attention (q / output ; k / v) | per block | iq2_s ; iq3_s |

| Layer norms + conv bases | 32 tensors | f32 (held, not quantized) |

The feed-forward block is 69% of the parameters, so it carries the size

savings; the selector and fc are kept higher-precision because they drive

which tokens the draft model proposes (acceptance), and the norms/conv-bases stay full

precision.

Required build: DFlash 2 support

This drafter needs a llama.cpp build with DFlash 2 support (merged into main on 2026.08.27).

Saving memory elsewhere: multimodal projector (mmproj)

If you are also looking to save memory on the vision side, try my Qwen3.8-27B-mmproj-Q5_K-MIX

projector checkpoint: 37% of the size of the upstream BF16 mmproj (331 MiB

vs 888 MiB) at a measured accuracy cost within the evaluation's noise

(see its model card for the full numbers).

Usage

Install llama.cpp with DFlash 2 support, then serve with this checkpoint as the draft:

llama-server \
  -hf <your-target-repo>/Qwen3.8-27B-GGUF:<target-file> \
  -hfd HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF:Q2_K_S \
  --spec-type draft-dflash \
  --spec-draft-n-max 3

---

Qwen3.8-27B-DFlash2-GGUF

Blog | GitHub

This repository contains GGUF conversions of

incoai/Qwen3.8-27B-DFlash2,

the DFlash 2 draft model for

Qwen/Qwen3.8-27B.

It is not a standalone language model: it runs inside a speculative

decoding server and drafts tokens for the target model to verify. The

checkpoints are also mirrored at

z-lab/Qwen3.8-27B-DFlash2-GGUF.

DFlash 2 is a block-diffusion drafter for speculative decoding. It predicts

a whole block of tokens in a single pass and keeps the top candidates at

every position. A lightweight selector then traces one coherent path through

them. Two-tap dynamic convolutions in the backbone keep the draft from

decaying toward the end of the block. Decoding is lossless: greedy output

matches the target model exactly, and sampling preserves its distribution.

<div align="center">

<img src="assets/dflash2-figure.png" alt="DFlash 2: parallel block drafting with a candidate path selector" width="100%">

</div>

| File | Size |

| :--- | ---: |

| Qwen3.8-27B-DFlash2-Q4_K_M.gguf | 1.1 GB |

| Qwen3.8-27B-DFlash2-Q8_0.gguf | 2.0 GB |

| Qwen3.8-27B-DFlash2-BF16.gguf | 3.8 GB |

Quick Start

Build llama.cpp with DFlash 2

support (PR #27342):

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

Then serve:

./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7

See the blog post for other engines and

more details.

Evaluation

• Target: ggml-org/Qwen3.8-27B-GGUF, Q4_K_M
• Sampling: Qwen3.8's officially recommended parameters (temperature 1.0, top-p 0.95, top-k 20), with xhigh reasoning effort
• Maximum new tokens: 2048
• Prompts: the first eight GSM8K test examples

Acceptance Length

Acceptance length is the per-request mean of completion tokens divided by

verification steps. Higher is better.

| Draft GGUF | Acceptance Length |

| :--- | ---: |

| BF16 | 5.28 |

| Q8_0 | 5.13 |

| Q4_K_M | 5.39 |

Full evaluations of the base checkpoint are on the

main model card.

Citation

If you find DFlash 2 useful, please cite:

@misc{inco2026dflash2,
  title  = {{DFlash 2: Keep Drafting Parallel}},
  author = {{Inco AI}},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}

Please also cite the original DFlash paper:

@inproceedings{chen2026dflash,
  title     = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author    = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}