---

license: mit

base_model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

base_model_relation: quantized

pipeline_tag: image-text-to-text

library_name: gguf

tags:

- gguf

- llama.cpp

- imatrix

- multimodal

- vision

- deepseek-v4

---

DeepSeek-V4-Flash-Vision-Exp — GGUF (multimodal, imatrix)

Imatrix GGUFs of deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

that actually do vision — four rungs from 66.9 to 108.7 GiB. It also ships the

importance matrix all four rungs were built with.

> ### ⚠️ Build llama.cpp from master — 2026-09-02 or later

>

> Upstream support landed in two halves on 2026-09-02:

> #28133 (vision tower) and

> #28154 (image-token routing).

> You need both. Release build b10763 carries only the first and will refuse to

> load these files; build master at or after

> 9400c894.

> No patch is needed any more — see Why a recent build.

Files

Four rungs, all built from the same importance matrix, which is included.

| rung | size | bits/weight¹ | notes |

|---|---:|---:|---|

| IQ1_M | 66.9 GiB | ~1.9 | smallest; most aggressive, least validated |

| IQ2_XXS | 78.8 GiB | ~2.2 | fits one 96 GB card at full 1M context |

| IQ2_S | 95.8 GiB | ~2.7 | |

| IQ3_XXS | 108.7 GiB | ~3.1 | highest quality here |

Shard naming is DeepSeek-V4-Flash-Vision-Exp-<RUNG>-0000{1..4}-of-00004.gguf.

| also included | size | |

|---|---:|---|

| mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf | 890 MiB | vision tower — required for images, works with any rung |

| DeepSeek-V4-Flash-Vision-Exp-imatrix.gguf | 470 MB | the importance matrix all four were built with |

¹ Effective average over the whole file, not the expert type alone. The rung name is

the dominant expert type; every rung mixes several — see Recipes. Note

general.file_type reads Q6_K in all of them: that field records the base type passed

to llama-quantize, and cannot express per-tensor overrides.

Will this run on my GPU?

Weights only — add ~13 GiB for KV/compute at a full 1M context, or far less at

shorter contexts. Keep the vision tower on CPU with --no-mmproj-offload unless you

have room; it is a one-off cost per image (466M params over ≤384 visual tokens).

| your VRAM | rung |

|---|---|

| 96 GB (1×) | IQ2_XXS at full 1M ctx (measured 96.9 GB peak), or IQ2_S/IQ3_XXS at reduced ctx |

| 80 GB (1×) | IQ1_M, or IQ2_XXS with a reduced --ctx-size |

| 64 GB (1×) | IQ1_M at reduced ctx |

| 128–192 GB (2×) | IQ3_XXS comfortably |

| less, or CPU | any rung with --n-cpu-moe / -ngl tuning; CPU-only works, slowly |

Build

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
# vision support merged 2026-09-02; no tagged release contains all of it yet,
# so build master (or check out 9400c894 or later explicitly)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-mtmd-cli

Run

Substitute your chosen rung for IQ2_XXS below.

./build/bin/llama-server \
  --model DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf --no-mmproj-offload \
  -ngl 999 --ctx-size 1048576 -fa on -b 2048 -ub 512 -np 1 --no-mmap --jinja

-ub 512 is needed to reach 1M context on a 96 GB card. -ub 2048 roughly doubles

prefill but adds ~4.2 GB of compute buffer, which will not fit alongside a speculative

draft; drop the draft if you want the larger ubatch.

Quick check with llama-mtmd-cli:

./build/bin/llama-mtmd-cli \
  -m DeepSeek-V4-Flash-Vision-Exp-IQ2_XXS-00001-of-00004.gguf \
  --mmproj mmproj-DeepSeek-V4-Flash-Vision-Exp-F16.gguf \
  --image photo.jpg -p "What is in this image?"

Why a recent build

DeepSeek-V4-Flash-Vision carries a second MoE router bias, ffn.gate.bias_vl,

used only for image tokens. The reference implementation selects experts with

scores + torch.where(input_ids >= vocab_size, bias_vl, bias)

and on the three hash-routed layers image tokens abandon the tid2eid lookup entirely

for a learned top-k over scores + bias_vl.

That is the whole reason for the version floor, and it shows up as two distinct

failures on an older build:

1. The tensor is not in the architecture yet. These files keep it, as

blk.N.exp_probs_b_vl.bias, so any llama.cpp before #28154 reports

wrong number of tensors; expected 1371, got 1328 and refuses to load. Note that

converters which simply drop bias_vl load anywhere — and route image tokens by

text preferences. Measured across all 43 layers the two biases are uncorrelated

(mean Pearson r = −0.026), so they encode genuinely different expert choices.

1. #28133 on its own aborts on the first image. An image arrives as an embeddings

batch, and llm_graph_input_embd::set_input only writes the token tensor when

ubatch->token is set — so the hash layers gather tid2eid[t_inp_tokens] from

uninitialised memory and hit GGML_ASSERT(i01 >= 0 && i01 < ne01). #28133 touches

no file under src/; #28154 is the half that fixes it.

#28154 adds FFN_EXP_PROBS_B_VL to the deepseek4 architecture and selects the bias

per batch. Because llama.cpp delivers an image as its own homogeneous embeddings batch —

every position in it is an image token, markers included — that is exact for the

batches mtmd produces, not an approximation.

Recipes

All four share the same non-expert treatment: attention, shared experts and dense at

Q6_K; token_embd and output at Q8_0; routers (ffn_gate_inp, exp_probs_b,

exp_probs_b_vl) at BF16/F32. They differ only in the routed experts:

| rung | ffn_gate_exps, ffn_up_exps | ffn_down_exps blk 0–9, 33–42 | ffn_down_exps blk 10–32 |

|---|---|---|---|

| IQ1_M | IQ1_M | IQ2_XS | IQ2_XXS |

| IQ2_XXS | IQ2_XXS | IQ3_XXS | IQ2_XS |

| IQ2_S | IQ2_S | IQ3_S | IQ3_XXS |

| IQ3_XXS | IQ3_XXS | IQ3_S | IQ3_S |

ffn_down gets the higher rung on the first ten and last ten blocks because that is

where quantization damage concentrates.

The importance matrix covers 606,208 tokens (74 × 8192-token chunks) with **full expert

coverage** — no expert went uncalibrated, which matters for a 256-expert MoE.

FP8 source weights were dequantized to BF16, not Q8_0 — BF16 represents the E4M3

significand exactly once the power-of-two E8M0 block scale is applied, so this avoids a

second lossy step on 365 attention and shared-expert tensors.

Known limitations

• In-span attention visibility is brand new and unmeasured here. The reference lets

tokens inside an [IMAGE_START, IMAGE_END] span see each other bidirectionally and

feeds that to the DFlash sparse-attention indexer. Upstream implements it in #28154

(swa_full_non_causal), which landed after the throughput figures below were taken

on a build that fell back to the causal window. Larger images, multiple images and

long contexts remain unmeasured either way.

• No KL-divergence ladder has been measured. The rungs are ordered by bits, which is

the expected ordering, but this repo makes no measured quality claim between them.

IQ1_M in particular is aggressive and unvalidated behaviourally.

• Testing so far is single-image. Multi-image and interleaved prompts are untested.
• This is an experimental upstream model, and its llama.cpp support merged on

2026-09-02. Expect the runtime side to keep moving.

Verification

**Re-checked 2026-09-02 against stock llama.cpp master (9cc33944f) with no patch of

any kind.** Both locally-held rungs answer image prompts correctly, on the CPU backend

and with CUDA offload. Prompts used DeepSeek's own example images.

| rung | image | answer |

|---|---|---|

| IQ2_XXS | carrots.jpeg | "Four orange carrots with green stems are arranged in a pile against a white background." |

| IQ2_XXS | corn.jpeg | "This image displays three ears of fresh corn on a plain white background, with one ear of corn partially husked to reveal its bright yellow kernels." |

| IQ1_M | carrots.jpeg | "The image depicts a group of four fresh, orange carrots with green stems, arranged in a pile on a white background." |

Throughput was measured earlier on a single RTX PRO 6000 Blackwell (96 GB), fully

resident at 1M context, with a Q2_K DSpark speculative draft — on the pre-merge patched

build, so treat it as indicative rather than a measurement of the code above:

| profile | prefill | decode |

|---|---:|---:|

| image, -ub 512, Q2_K DSpark draft | 135.8 tok/s | 129.4 tok/s (63.9% draft acceptance) |

| text, same | 177.9 | 140.9 (68.2%) |

License

MIT, inherited from the base model.