---

base_model:

• deepseek-ai/DeepSeek-V4-Flash-0731

base_model_relation: quantized

language:

• en

license: mit

library_name: gguf

pipeline_tag: text-generation

tags:

• deepseek-v4
• deepseek-v4-flash
• gguf
• ds4
• imatrix
• mlx
• apple-silicon
• m5-max
• 128gb
• mixture-of-experts
• quantized

---

DeepSeek-V4-Flash-0731 — Mixed Q2/Q4 GGUF for ds4 (128 GB Macs)

Mixed-precision GGUFs of DeepSeek-V4-Flash-0731, built for

antirez/ds4 (DwarfStar) on 128 GB Apple Silicon.

Reproduces the canonical antirez "layers 37-42 Q4 / everything else Q2" recipe,

applied to the 0731 weights.

The file

DeepSeek-V4-Flash-0731-Layers37-42Q4K-mixed-realimatrix-v2.gguf — 91 GiB

(97,426,596,736 bytes), quantized with a real activation imatrix.

An earlier build of this same recipe used the quantizer's synthetic

weight-energy fallback instead. It has been removed in favour of this one, which

measures better on every metric tested (see Quality below). The two had an

identical tensor-type layout and identical runtime cost — they differed only in

which quantization codes were chosen for the routed experts.

Why this exists

deepseek-ai/DeepSeek-V4-Flash-0731 ships as ~155 GB of fp8 safetensors — too

large for a 128 GB Mac alongside KV cache and the OS. These files compress the

0731 snapshot to ~91 GB using antirez's mixed-precision strategy, so they drop

straight into an existing ds4 setup.

Quantization recipe

| Tensors | Quant | Rationale |

|---|---|---|

| Routed MoE experts, layers 37–42 (the "hot" layers) | Q4_K | Higher fidelity where it matters most |

| Routed MoE experts, all other layers | IQ2_XXS (~2-bit) | The cold bulk, aggressively compressed |

| Down projections | Q2_K | 2-bit |

| Attention projections | Q8_0 | Small, fires every token — keep sharp |

| Shared experts | Q8_0 | Always-on dense path |

| Output head | Q8_0 | Critical, small |

Verified layout: 18 × Q4_K + 37 × Q2_K + 74 × IQ2_XXS = 129 routed-expert

tensors across 43 layers.

v2 is built in a single deepseek4-quantize pass, using --tensor-type

overrides to promote layers 37-42 rather than quantizing a separate ~153 GiB Q4

donor and splicing it in:

args=()
for L in 37 38 39 40 41 42; do
  for T in gate down up; do
    args+=(--tensor-type "blk.$L.ffn_${T}_exps.weight=q4_k")
  done
done

gguf-tools/deepseek4-quantize \
  --hf   /path/to/DeepSeek-V4-Flash-0731 \
  --template DeepSeek-V4-Flash-0731-Q2-imatrix.gguf \
  --imatrix  DeepSeek-V4-Flash-0731-routed-moe-real.dat \
  --out  DeepSeek-V4-Flash-0731-Layers37-42Q4K-mixed-realimatrix-v2.gguf \
  "${args[@]}" --threads 16

This produces a byte-identical layout to the splice route (same 1328 tensors,

same 18 type changes) while skipping an entire quantize pass.

The imatrix, and a trap worth documenting

v2 uses a real 0731-native routed-MoE imatrix collected with

ds4 --imatrix-out (129 entries = 43 layers × {gate, down, up}, 2752 chunks —

comparable calibration depth to antirez's own reference imatrix). v1 used the

quantizer's synthetic weight-energy fallback.

Do not pass --imatrix-strict with a routed-MoE imatrix. The first v2 attempt

did, and produced a silently truncated 73 GiB file. The flag means *"fail if a

quantized tensor has no matching imatrix vector"* — but a routed-MoE imatrix by

construction only covers routed experts. It therefore exit(1)s on

token_embd.weight, the first non-expert tensor in write order, at byte offset

77,922,699,264 (72.6 GiB), killing the process mid-write. This is deterministic:

strict mode can never succeed with a routed-MoE imatrix. Plain --imatrix is

correct usage — the fallback for non-expert tensors is the intended path, not

silent data loss.

Two related gotchas:

• Template must match the revision. Templating off the older chat-v2 GGUF

instead of the 0731 Q2 shifts the planned size by ~165 MB.

• The tool prints decimal bytes; ls -lh prints GiB. A correct file is

97,426,596,736 B, which ls -lh shows as 91G. A real-imatrix build is also

exactly 288 bytes larger than a non-imatrix one — those are the four

quantize.imatrix.* provenance keys, not corruption.

Quality: real imatrix vs the synthetic build it replaced

Scored with gguf-tools/quality-testing/score_official against the **100 tracked

DeepSeek API continuations** in data/flash, compared via compare_scores.py.

The metric is target-token negative log likelihood — how much probability each

GGUF assigns to the official continuation, token by token. This avoids judging

quality from a single sampled answer.

| Metric | v1 (synthetic) | v2 (real imatrix) |

|---|---|---|

| Mean NLL (lower is better) | 0.564260 | 0.522266 (−7.44%) |

| Cases won (of 100) | 38 | 62 (0 ties) |

| First-token match | 44/100 | 53/100 |

| Avg greedy longest-common-prefix | 4.770 | 5.490 tokens |

All four metrics move the same direction, which is stronger evidence than the

mean alone — a mean shift can come from a couple of outliers, but a 62/38 case

split with better first-token agreement and longer greedy prefixes indicates a

broad improvement.

Honest caveat on strength of evidence: the 62/38 split is exact-binomial

p = 0.0105 — significant at p < 0.05, but not at p < 0.01, and this is a

single 100-case run. Treat it as a real and consistent improvement, not a

dramatic one. v2 still loses 38 cases; it wins more of them and loses them by

less.

Hardware / performance

Tested on Apple M5 Max, 128 GB unified memory:

• Model maps to ~92.9 GB; loads with comfortable headroom for KV cache
• ~34.7 tokens/sec generation, ~23–58 tokens/sec prefill (ds4, Metal)
• Native ds4 --ctx 200000 works

Throughput figures were measured on v1. v2 has an identical tensor-type

histogram, so runtime cost is unchanged — the imatrix affects which codes are

chosen, not how many bits are read per weight.

Usage

# ds4 (DwarfStar) — intended runtime
./ds4 -m DeepSeek-V4-Flash-0731-Layers37-42Q4K-mixed-realimatrix-v2.gguf --ctx 200000

# As a drop-in for an existing ds4 install:
ln -sfn /path/to/this/file.gguf ds4flash.gguf
./ds4-server --ctx 200000

Not a general GGUF. Like all antirez-recipe files, this requires the ds4

engine's specific tensor layout — it will not load in a generic llama.cpp/Ollama

build.

Comparison vs prior revision (0730)

Basic reasoning (greedy, temp=0) is equivalent on math, coding, and knowledge

prompts. The 0731 finetune shows a more verbose structured "show your work"

style. Across four agentic tasks via ds4-agent (file read, CSV filtering, log

scanning, and an implement→write→run→verify loop), 0730 and 0731 were

equivalent — 0731's distinguishing behavior was more transparent self-monitoring

and error recovery rather than flawless execution. A rigorous multi-turn agent

benchmark would be needed to confirm 0731's "enhanced agentic capabilities"

claim; that probe only established parity at the basic-to-intermediate level.

Known caveats

• DSpark/MTP speculative decoding: 0731 adds dspark_* config fields. Base

inference works fully; whether ds4's MTP path exploits 0731's DSpark config is

unverified. The speculative-decoding speedup may not transfer without engine

work or a DSpark-support companion GGUF.

• Built and tested on M5 Max only; behavior on other Apple Silicon (M3/M4) is

expected to be equivalent but not separately validated.

• Quality evaluation is a single 100-case NLL run (see caveat above).

Acknowledgements

Recipe and tooling (deepseek4-quantize, the quality-testing harness, the

mixed-precision strategy) are entirely from

antirez/ds4. This is a community quant applying

that tooling to the 0731 weights, not an official DeepSeek or antirez release.