---

quantized_by: tarruda

pipeline_tag: image-text-to-text

base_model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

base_model_relation: quantized

license: mit

license_link: https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md

tags:

• gguf
• imatrix
• conversational
• multimodal
• vision
• deepseek
• llama.cpp

---

DeepSeek V4 Flash Vision Exp GGUF

WARNING: This is not currently supported by llama.cpp master and requires my experimental branch to use https://github.com/tarruda/llama.cpp/tree/dsv4-vision-exp-support

GGUF quantizations for deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.

DeepSeek published the original model weights in MXFP4, so the MXFP4 GGUFs in

this repo are direct conversions of those original safetensors.

Quant Recipes

| Recipe | Quant size | Default type | Expert tensor type |

| --- | ---: | --- | --- |

| Q2_K | 95464.96 MiB (2.82 BPW) | Q8_0 | Q2_K |

| IQ3_XXS | 109912.96 MiB (3.24 BPW) | Q8_0 | IQ3_XXS |

| Q3_K | 122296.96 MiB (3.61 BPW) | Q8_0 | Q3_K |

For every quant recipe, ffn_down_exps, ffn_gate_exps, and ffn_up_exps use the recipe's expert type. Compressor, indexer, token embedding, output, hyper-connection, router-bias, and token-to-expert mapping tensors retain the BF16, F32, or I32 types specified in scripts/quantize.sh. All three quants use the included 726-entry, 812-chunk importance matrix.

Usage

model="./IQ3_XXS/DeepSeek-V4-Flash-Vision-Exp-IQ3_XXS-00001-of-00004.gguf"
mmproj="./mmproj-DeepSeek-V4-Flash-Vision-Exp-BF16.gguf"

ctx=262144
parallel=1

llama-server --no-mmap --no-warmup \
  --model "$model" --mmproj "$mmproj" \
  ---ctx-size $ctx_size -np $parallel \
  --repeat-penalty 1.0  --presence-penalty 0.0 \
  --chat-template-kwargs '{"reasoning_effort": "max"}' \
  --reasoning-preserve \
  -b 2048 -ub 2048 \
  --cache-ram 1024 -ctxcp 12 \
  --temp 1.0 --top-p 1.0