---

base_model:

• deepseek-ai/DeepSeek-V4-Flash

---

DeepSeek-V4-Flash GGUF quantizations

High-quality GGUF quantizations of DeepSeek-V4-Flash, produced with llama.cpp.

Important model detail

The source HF model is already natively quantized:

• Dense weights: fp8, converted to Q8_0/BF16 in the reference GGUF
• Routed MoE experts: fp4 MXFP4

Most parameters are in the routed experts, so normal Q4_K_M is not a useful upload target here. It is only marginally smaller than the reference, and it was dominated by better measured options. The final set below keeps only the quants that make practical sense.

Final files

| File | Size | BPW | KLD | Notes |

|------|------|-----|-----|-------|

| DeepSeek-V4-Flash.IQ2_S.gguf | 82.0 GiB | 2.48 | 0.151700 | Smallest option. Experts use the IQ2_S recipe; dense tensors are protected at Q8_0. |

| DeepSeek-V4-Flash.IQ3_XXS-AS.gguf | 95.4 GiB | 2.88 | 0.111648 | Asymmetric small-mid option. Down/up experts use iq3_xxs; gate experts use iq2_xxs. |

| DeepSeek-V4-Flash.IQ3_XXS.gguf | 106.1 GiB | 3.21 | 0.077500 | Standard 3-bit option. All experts use iq3_xxs; dense tensors are protected at Q8_0. |

| DeepSeek-V4-Flash.IQ3_XXS-L21.gguf | 124.8 GiB | 3.77 | 0.053911 | High-quality compressed option. Final 21 expert layers stay native MXFP4; earlier experts use iq3_xxs. |

| DeepSeek-V4-Flash.IQ3_XXS-L39.gguf | 140.9 GiB | 4.26 | 0.043924 | Near-reference compact option. Final 39 expert layers stay native MXFP4; first four expert layers use iq3_xxs. |

| DeepSeek-V4-Flash.IQ4_XS.gguf | 144.4 GiB | 4.36 | 0.038715 | Best non-reference standard quant found. Experts use IQ4_XS; dense tensors are protected at Q8_0. |

| DeepSeek-V4-Flash.Q8_0.gguf | 145.6 GiB | 4.40 | 0.000000 | Highest fidelity reference. Dense Q8_0/BF16 plus native MXFP4 experts. |

KLD is the mean KL divergence of each quantized model against the Q8_0/reference logits on a held-out evaluation split. Lower is better. The reference-vs-self KLD is effectively zero.

Recommendations

• Best quality: DeepSeek-V4-Flash.Q8_0.gguf. This is the exact GGUF reference.
• Best near-reference non-reference quant: DeepSeek-V4-Flash.IQ4_XS.gguf.
• Compact near-reference option: DeepSeek-V4-Flash.IQ3_XXS-L39.gguf.
• High-quality compressed option: DeepSeek-V4-Flash.IQ3_XXS-L21.gguf.
• Standard 3-bit option: DeepSeek-V4-Flash.IQ3_XXS.gguf.
• Smaller mid option: DeepSeek-V4-Flash.IQ3_XXS-AS.gguf.
• Smallest option: DeepSeek-V4-Flash.IQ2_S.gguf.

Measurement methodology

• Reference: converted GGUF with Q8_0/BF16 dense tensors and native MXFP4 experts.
• Calibration: imatrix computed on a training split disjoint from evaluation data.
• Evaluation: KLD measured only on the held-out evaluation split against the reference logits.
• Metric used for selection: KLD only.

Key findings

• Q4_K_M is not included because it was almost the same size as the reference and was dominated by IQ4_XS and IQ3_XXS-L39.
• iq3_xxs worked unusually well for these MXFP4-source expert tensors, beating larger nominal 3-bit variants in the tested recipes.
• Keeping later expert layers at native MXFP4 gives a strong size-quality ladder; L21 and L39 were kept as the most useful points from that ladder.
• IQ4_XS was the best standard near-reference quant found, with lower KLD than the removed Q4_K_M file.

Running

Example:

llama-cli -m DeepSeek-V4-Flash.IQ3_XXS.gguf -ngl 999 -fa on -c 8192

Flash attention (-fa on) is recommended when supported.

Reproducibility notes

The quantization recipes used llama.cpp llama-quantize --allow-requantize with imatrix guidance and tensor-type override files. The hash-routing tensor ffn_gate_tid2eid must stay i32. Small F32 tensors such as ape, hc_, attn_sinks, and output_hc_ must stay native for low-bit imatrix-requiring ftypes.

External Antirez comparison

The Antirez DeepSeek-V4-Flash GGUFs were also evaluated against the same Q8_0/reference KL base and held-out split. The MTP side GGUF was skipped because it is not a standalone full model.

!KLD vs size comparison with Antirez

| File | Size | KLD | Result |

|------|------|-----|--------|

| DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf | 80.8 GiB | 0.184799 | Slightly smaller than our IQ2_S, but higher KLD. |

| DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf | 80.8 GiB | 0.183665 | Slightly smaller than our IQ2_S, but higher KLD. |

| DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf | 90.9 GiB | 0.158371 | Dominated by our IQ2_S: larger and higher KLD. |

| DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf | 153.3 GiB | 0.035836 | Low KLD, but larger than the Q8_0 reference, so not a practical upload target. |

| DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.gguf | 153.3 GiB | 0.035170 | Low KLD, but larger than the Q8_0 reference, so not a practical upload target. |

Combined graphics: kld_size_comparison_with_antirez.png and kld_size_comparison_with_antirez.svg.