bartowski/DeepSeek-V4-Flash-GGUF overview

---

quantized_by: bartowski

pipeline_tag: text-generation

license: mit

base_model_relation: quantized

base_model: deepseek-ai/DeepSeek-V4-Flash

---

Llamacpp Quantizations of DeepSeek-V4-Flash by deepseek-ai

Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggml-org/llama.cpp/releases/tag/b9843">b9843</a> for quantization.

Original model: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash

This model is in MXFP4 and as such has only been provided in MXFP4 format!

No other sizes can be provided unfortunately as MXFP4 does not quantize properly.

Run in your choice of tools:

• llama.cpp
• ramalama
• LM Studio
• koboldcpp
• Jan AI
• Text Generation Web UI
• LoLLMs
• Atomic Chat

Note: since it's a newly supported model, you may need to wait for an update from the developers.

Prompt format

No prompt format found

Download the MXFP4 files:

| Filename | Quant type | File Size | Split | Description |

| -------- | ---------- | --------- | ----- | ----------- |

| DeepSeek-V4-Flash-MXFP4.gguf | MXFP4 | 156.00GB | true | Original quality. |

Downloading using huggingface-cli

<details>

<summary>Click to view download instructions</summary>

First, make sure you have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

huggingface-cli download bartowski/DeepSeek-V4-Flash-GGUF --include "DeepSeek-V4-Flash-MXFP4*" --local-dir ./

You can either specify a new local-dir (DeepSeek-V4-Flash-MXFP4) or download them all in place (./)

</details>

Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.

Thank you ZeroWw for the inspiration to experiment with embed/output.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski