---

license: mit

library_name: gguf

pipeline_tag: text-generation

base_model: deepseek-ai/DeepSeek-V4-Flash

base_model_relation: quantized

quantized_by: antirez

language:

- en

tags:

- gguf

- quantized

- deepseek

- deepseek-v4

- deepseek-v4-flash

- moe

- mixture-of-experts

- 2-bit

- 4-bit

- iq2_xxs

- q2_k

- q4_k

- ds4

- apple-silicon

- metal

---

DeepSeek V4 Flash — GGUF for ds4

This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).

https://github.com/antirez/ds4

Files

| File | Size | Routed experts (ffn_{gate,up,down}_exps) | Everything else |

|---|---:|---|---|

| DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf | 80.8 GiB | IQ2_XXS (gate, up) + Q2_K (down) | Q8_0 attn proj / shared experts / output, F16 router + embed + indexer + compressor + HC, F32 norms / sinks / bias |

| DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf | 153.3 GiB | Q4_K (all three) | same as above |

| DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf | 3.6 GiB | MTP / speculative-decoding support (optional, not standalone). | |

Use q2 on 128 GB Mac machines, q4 on machines with ≥ 256 GB RAM, pair either with MTP for optional speculative decoding.

Quantization recipe

The filename is the spec. In detail, for the q2 file:

| Tensor class | Quant | Notes |

|---|---|---|

| blk..ffn_gate_exps, blk..ffn_up_exps | IQ2_XXS | routed-expert up/gate |

| blk.*.ffn_down_exps | Q2_K | routed-expert down (K-quant for quality) |

| blk.*.ffn_{gate,up,down}_shexp | Q8_0 | shared experts |

| blk.*.attn_q_a, attn_q_b, attn_kv, attn_output_a, attn_output_b | Q8_0 | all attention projections (MLA + low-rank output) |

| output.weight | Q8_0 | output head |

| token_embd.weight | F16 | input embedding |

| blk.*.ffn_gate_inp (router) | F16 | learned router |

| blk..exp_probs_b (router bias), blk..attn_sinks, all *_norm.weight | F32 | |

| blk.*.ffn_gate_tid2eid | I32 | hash-routing tables (first 3 layers only) |

| blk..attn_compressor_, blk..indexer_, blk..hc_, blk..output_hc_ | F16 / F32 | DSv4-specific auxiliary blocks |

For the q4 file, only the three routed-expert classes change to Q4_K. Everything else is byte-for-byte identical to the q2 recipe.

The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on them costs less in average quality than the same treatment of router, projections, or shared experts. Keeping the decision-making components at Q8_0 preserves model behavior; crushing the experts buys the size.

Usage

git clone https://github.com/antirez/ds4
cd ds4
./download_model.sh q2     # 128 GB RAM machines
./download_model.sh q4     # >= 256 GB RAM machines
./download_model.sh mtp    # optional MTP / speculative decoding
make

./ds4 -p "Explain Redis streams in one paragraph."
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

The download_model.sh script fetches from this repository, resumes partial downloads, and points ./ds4flash.gguf at the selected variant.

License

MIT. The base model copyright is held by DeepSeek; the GGUFs are redistributed under the base model's release terms.