---

language:

• en
• code

base_model:

• unsloth/DeepSeek-V4-Flash-0731-GGUF

tags:

• speculative-decoding
• draft-model
• gguf
• quantization

---

DeepSeek-V4-Flash-0731 DSpark Drafter Q2_K_S

Optimized draft model for speculative decoding with DeepSeek-V4-Flash-0731. Quantized to Q2_K_S with minimal quality loss.

Performance (AMD Strix Halo)

Full speculative-decoding benchmark, target model

DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf (96 GB, IQ3_XXS + Q6 attention),

7 task categories × 6 draft lengths × 5 runs each (210 requests per draft model).

Sampling: temp 0.9 / top_p 0.95 / min_p 0.01, ctx 65536, reasoning_effort=low.

Baseline (no draft model): 20.48 tok/s.

Headline numbers

| Metric | Q8_0 draft | Q2_K_S draft (this) |

| -------- | ----------- | --------------------- |

| File size | 10.15 GB | 6.45 GB (−36%) |

| Best overall decode | 28.43 tok/s | 28.58 tok/s |

| Best overall speedup vs baseline | 1.39× | 1.40× |

| Optimal n_max (overall) | 3 | 3 |

Q2_K_S matches the Q8_0 drafter within noise (±1–3% on every n_max and every

category, tie per the benchmark's own threshold) while being 36% smaller than

Q8_0 and 39% smaller than the BF16 drafter (10.54 GB).

Speedup vs baseline (20.48 tok/s) by task category

Best result per category, averaged over Q8_0 + Q2_K_S drafts:

| Category | Best n_max | Decode tok/s | Speedup |

| ---------- | -----------: | -------------: | --------: |

| code | 3 | 28.99 | 1.42× |

| prose | 2 | 23.38 | 1.14× |

| translate | 2 | 25.09 | 1.22× |

| json | 3 | 28.59 | 1.40× |

| math | 3 | 31.22 | 1.52× |

| repeat | 5 | 40.17 | 1.96× |

| dialog | 3 | 25.48 | 1.24× |

| overall | 3 | 28.50 | 1.39× |

> Note on the averages. The 28.5 tok/s overall figure is pulled up by the mix: the suite is heavy on tasks that speculative decoding likes (repeat, math, code). Prompts also used a relatively short context (up to ~32k tokens of the 64k window). On a more typical mixed workload — especially with a fuller context — expect closer to 22–28 tok/s.

n_max sweep (average of Q8_0 + Q2_K_S)

| n_max | Decode tok/s | Speedup | Accept rate |

| ------: | -------------: | --------: | ------------: |

| 2 | 27.12 | 1.32× | 0.693 |

| 3 | 28.50 | 1.39× | 0.602 |

| 4 | 27.73 | 1.35× | 0.522 |

| 5 | 26.43 | 1.29× | 0.455 |

| 6 | 26.40 | 1.29× | 0.456 |

| 7 | 26.52 | 1.29× | 0.451 |

n_max=3 is the sweet spot: longer drafts (n_max≥5) buy almost nothing and even

start to hurt on prose/translate/dialog (low acceptance → wasted draft tokens).

Practical recommendation

Run the server with --spec-draft-n-max 3. If the workload is

code/json/math-heavy, n_max=3 is optimal; for repeat-like (high-acceptance)

loads, n_max=5–6 adds ~1.5× more. The Q2_K_S drafter is the drop-in choice:

same speed as Q8_0 at 36% smaller footprint. Treat the 28.5 tok/s headline as

an optimistic mix, not a day-to-day guarantee; 22–28 tok/s is the realistic band.

Charts

Full chart set in bench_reports/20260817_225557/charts/:

!Summary dashboard

!Best speedup per category

!Decode tok/s vs n_max (with baseline)

!Decode tok/s by category — average of Q8+Q2

!Speedup heatmap: category × n_max

!Speedup vs baseline by category

Also in the same directory: per-draft decode charts (decode_tps_by_category_q8.png,

_q2.png, overlay variant), per-draft heatmaps (speedup_heatmap_q8.png, _q2.png),

decode_tps_vs_nmax_avg.png, speedup_grouped_bars.png, speedup_vs_accept_rate.png.

Benchmark setup

• Baseline 20.48 tok/s measured without -md / --spec-type on the same target model
• Categories: code, prose, translate, json, math, repeat, dialog
• Server: llama-server from strix-halo-llamacpp, -fa 1, -ngld 999, -np 1, draft-dspark, ngram-mod off
• 5 runs per cell; reported values are averages with min–max spread

Quantization Details

Quantized using upstream llama.cpp llama-quantize with --allow-requantize and custom tensor-type overrides.

Expert weights (92% of model):

• gate/up/down_exps: Q2_K_S

Attention & shared experts:

• attn_kv, attn_output, attn_q_b: Q8_0
• attn_q_a: Q5_K
• ffn_*_shexp: Q5_K/Q6_K
• fc: Q6_K

Protected:

• MoE router, markov weights, hyper connections: F16/Q8_0
• All norms, sinks, biases: F32

Credits

• Base model: DeepSeek-V4-Flash-0731
• BF16 GGUF: unsloth/DeepSeek-V4-Flash-0731-GGUF
• Quantization: llama.cpp (upstream)
• Inference & benchmarking: strix-halo-llamacpp