---

license: mit

base_model: deepseek-ai/DeepSeek-V4-Flash

pipeline_tag: text-generation

tags:

• deepseek
• deepseek-v4
• gguf
• llama.cpp
• mtp
• speculative-decoding
• mixture-of-experts

library_name: llama.cpp

---

DeepSeek-V4-Flash MTP GGUF — the only GGUFs with the draft head intact

Every public DeepSeek-V4-Flash GGUF strips the model's multi-token-prediction (MTP)

head: llama.cpp's converter drops the mtp.* tensors and its C++ has no code to run

them. These are, as far as we know, the first and only V4-Flash GGUFs anywhere with

the head preserved and working — the model drafts a token ahead of itself and

verifies it in the same forward pass, for free decode speedup with zero quality

change (every draft is verified by the full model).

Two files:

| File | Size | Experts | MMLU¹ | Tool hardmode¹ | Decode² |

|---|---|---|---|---|---|

| Q8-MTP (the master — recommended) | 160 GB | native MXFP4 (original release bits, never requantized) | 88.3 | 43–73 (mean 55) | 26–27 tok/s |

| Q3_K_M-MTP (speed build) | 143 GB | Q3_K/Q4_K | 84.0 | 40–50 | 30.5 tok/s |

For reference, measured identically on the same hardware: the popular community

Q4_K_M-XL scores 85.0 MMLU at 16.5 tok/s from a 175 GB file, and Unsloth UD-IQ3_XXS

scores 79.6 at ~26 tok/s from 102 GB.

¹ MMLU = 120-question stratified sample; tool hardmode = tool-eval-bench

adversarial suite (15 scenarios; we publish the range over 3 runs because the suite

has ±10 pts of run-to-run variance — single-run scores you see elsewhere carry the

same error bar, unstated).

² Single stream, MTP n_max=1, on 4× RTX PRO 4500 Blackwell (128 GB VRAM) +

Threadripper 9970X, experts partially spilled to DDR5. Fully GPU-resident setups

should see substantially more MTP gain (we measured 2.2× on a zero-spill control).

Why the "Q8" master beats every Q4 quant

DeepSeek released V4-Flash with the routed experts already in MXFP4 (4-bit). Our

converter passes those bits through untouched and stores everything else (attention,

sparse-attention indexer, MTP head, norms) at Q8_0. A "Q4_K" quant of this model is a

sideways re-encode: it snaps weights that are already 4-bit onto a different 4-bit

grid — a lossy round-trip that also produces a bigger file (Q4_K spends 4.5 bpw vs

MXFP4's 4.25). Measured cost of that re-encode: 85.0 vs 88.3 MMLU, +15 GB.

Quantize downward to shrink (that's the Q3_K_M speed build). Never sideways.

Requirements

MTP for DeepSeek-V4 is not in upstream llama.cpp yet (the maintainers have

signaled native support is planned). Until that lands, build from our branch:

git clone --branch dsv4-mtp https://github.com/rogerai-fyi/llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server

The branch contains the converter patch, the MTP graph implementation for the

deepseek4 architecture, and three unrelated bug fixes it flushed out (SIGFPE on

empty KV-cache filters, quantizer crash on integer routing tensors, a

metadata-ordering trap). A rebased-on-master variant lives on the same fork as

dsv4-mtp-upstream.

Usage

# Q8-MTP master, 4×32 GB VRAM + DDR5 spill, MTP drafting on:
llama-server --model DeepSeek-V4-Flash-Q8-MTP-00001-of-00004.gguf \
  --n-gpu-layers 999 --n-cpu-moe 14 \
  --split-mode layer --tensor-split 20,8,8,8 \
  --no-repack --ubatch-size 128 \
  --flash-attn on --no-kv-unified --jinja --reasoning-format deepseek \
  --temp 1.0 --top-p 1.0 --min-p 0.0 \
  --spec-type draft-mtp --spec-draft-n-max 1

Notes from production (we serve the master daily on the box above):

• --spec-draft-n-max 1 is the sweet spot when experts spill to system RAM.

Verifying N drafted tokens costs ~N× the DDR5 expert reads, so long drafts lose;

a 1-token draft is accepted ~100% of the time (it is literally the head's training

objective). The less you spill, the higher you can push it.

• Use mmap (default) — --no-mmap needs host RAM larger than the file.
• --no-repack is required at high --n-cpu-moe counts on current builds.
• Adjust --n-cpu-moe / --tensor-split to your VRAM; the Q3_K_M build needs

--n-cpu-moe 10 --tensor-split 17,9,9,9 on the same hardware.

• KV cache stays f16: q8_0 KV corrupts V4 output (llama.cpp #24162). V4's sparse

attention keeps KV small anyway.

Provenance

Converted directly from deepseek-ai/DeepSeek-V4-Flash (FP8/MXFP4 release) with our

patched converter: mtp.* tensors rekeyed as layer 43, glue projections folded via

the e_proj(e) + h_proj(x) = eh_proj(concat[e;x]) identity, all seven glue tensors

verified against DeepSeek's reference implementation. The MTP head is a full extra

transformer layer (MLA attention, 256-expert MoE, hyper-connection mixers — 1,575

source tensors). Metadata (block_count=44, nextn_predict_layers=1) is baked in.

Full build story, benchmarks and the speculation-economics analysis:

https://rogerai.fyi/broadcasts-deepseek-mtp-gguf.html

DeepSeek-V4-Flash is MIT-licensed; these derivatives are too.

((•)) RogerAI — own your GPU. curl -fsSL https://rogerai.fyi/install.sh | sh