---

base_model: antirez/deepseek-v4-gguf

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

license: mit

language:

- en

tags:

- deepseek

- deepseek-v4-flash

- gguf

- quantized

- imatrix

- mixture-of-experts

- apple-silicon

- metal

- m5-max

- ds4

---

DeepSeek-V4-Flash — M5 Max mixed-quant GGUF

Custom mixed-precision, imatrix-calibrated GGUF quantizations of

DeepSeek-V4-Flash,

tuned to run resident at long context on a single Apple M5 Max (128 GB).

The quant recipe is not uniform: bit-width is assigned per tensor role and per layer to spend

bits where they buy the most quality, and the low-bit routed experts use the M5 **Neural

Accelerators** at inference time.

> [!IMPORTANT]

> These GGUFs target the ds4 (DwarfStar) engine, not llama.cpp. They use the deepseek4

> architecture (MLA-style attention + compressed/indexed KV + hyper-connections) and a

> mixed per-layer quant layout that mainline llama.cpp does not implement. Loading them in

> llama.cpp/Ollama/LM Studio will not work. See Usage for how to run them.

---

Model at a glance

| | |

|---|---|

| Base model | DeepSeek-V4-Flash (arch deepseek4) |

| Layers | 43 |

| Routing | 256 routed experts, top-6 active + 1 shared per token |

| Hidden size | 4096 |

| Attention heads | 64 (MLA + compressed/indexed KV) |

| Vocab | 129,280 (GPT-2 BPE) |

| Native context | 1,048,576 (1M) |

| Calibration | importance-matrix (imatrix) |

---

Files

| File | Size | Purpose |

|---|---:|---|

| DeepSeek-V4-Flash-Layers1-2.37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf (deg8) | ~94 GiB | The model. Mixed-quant, imatrix-calibrated. |

| DeepSeek-V4-Flash-DSpark-support.gguf | ~5.6 GiB | Optional DSpark speculative-decode drafter. Load with --mtp ... --dspark for faster greedy/low-temp code generation. Not needed for plain decoding. |

Attention projections, the shared expert, and the output head are Q8_0; hyper-connection /

compressor / indexer tensors and norms stay F16/F32.

deg8 tensor layout

• Routed experts, layers 1-2 & 37-42 (8 layers): gate / up / down = Q4_K
• Routed experts, all other layers (35 layers): gate / up = IQ2_XXS, down = Q2_K
• Attention q/kv/o projections, shared expert, output head: Q8_0
• HC / compressor / indexer / norms: F16 / F32

Measured tensor-type histogram (deg8): F32 ×492, F16 ×359, Q8_0 ×345, IQ2_XXS ×70, Q2_K ×35,

Q4_K ×24, I32 ×3 (1328 tensors total).

Layers 1-2 are promoted to Q4_K because, on this model, those two blocks route almost all tokens

to a single expert — spending the extra bits there is unusually cost-effective.

---

Quality

Scored with the engine's teacher-forced next-token NLL harness (100 held-out continuations,

ctx 4096, greedy): deg8 avg NLL ≈ 0.329 (lower is better).

For context, a leaner recipe that promotes only layers 37-42 to Q4_K scores ~0.347 — so the two

extra promoted blocks in deg8 buy ~5% lower NLL for ~3 GiB, which is why deg8 is the shipped

default.

---

Performance (Apple M5 Max, 128 GB, ds4 Metal backend)

Measured with fans pinned and drift-cancelling interleaved runs. Numbers are **engine- and

hardware-specific** and will not transfer to other runtimes.

| Metric | deg8 |

|---|---:|

| Prefill (IQ2 expert layers, fused Neural-Accelerator kernel, ctx 8192) | ~480 tok/s |

| Decode, plain greedy | ~40 tok/s |

| Decode, DSpark speculative (direct code, --nothink) | ~45-48 tok/s |

| Verified context with the DSpark support model resident | 192k |

The prefill figure reflects ds4's fused matmul2d pair-SwiGLU kernel running the compute-bound

IQ2/Q4_K routed experts on the M5 Neural Accelerators. Decode is memory-bandwidth bound, so it

tracks the resident-weight bytes per token.

---

Hardware requirements

• Apple M5 Max, 128 GB unified memory (the Neural-Accelerator prefill path needs Apple10 GPU

family; other Apple Silicon will run the simd fallback).

• deg8 needs ~94 GiB resident. At 192k context with the speculative support model, use

--prefill-chunk 2048 to stay inside the ~121.6 GiB Metal working-set budget.

---

Usage

Run with the ds4 / DwarfStar engine:

# interactive coding agent, long context, speculative decode
ds4-agent \
  -m DeepSeek-V4-Flash-Layers1-2.37-42Q4KExperts-...-imatrix.gguf \
  --mtp DeepSeek-V4-Flash-DSpark-support.gguf --dspark \
  --temp 0.6 --top-p 0.95 \
  --ctx 192000 --prefill-chunk 2048 --tokens 12288 --threads 12

# one-shot
ds4 -m DeepSeek-V4-Flash-Layers1-2.37-42Q4KExperts-...-imatrix.gguf \
    -p "Explain the CAP theorem." -n 512 --temp 0

DSpark speculative decode is optional (adds a ~5.7 GiB support GGUF); it speeds up direct code

generation and is lossless at any temperature, but is a slight net loss on heavy chain-of-thought

prose — drop --dspark there.

---

Quantization method

Produced by a source-free GGUF→GGUF requantization: tensors are re-quantized directly from a

higher-precision GGUF (no PyTorch checkpoint required), guided by an importance matrix, with the

per-tensor / per-layer bit assignment above. The Q8_0 attention and shared-expert projections are

near-lossless; the routed-expert tail carries the aggressive low-bit quant because it is only a

small fraction of the per-token bandwidth.

Limitations

• ds4-only — not compatible with llama.cpp and its downstream tools.
• The aggressive IQ2_XXS routed experts trade some quality for footprint; for maximum fidelity use

the full-precision base model instead.

• Benchmarks are single-machine (one M5 Max); your throughput depends on thermal state and

concurrent load.

Credits & license

Released under the MIT license. The base model copyright is held by DeepSeek; these GGUFs

are redistributed under the base model's release terms.

• Base model: DeepSeek-V4-Flash by DeepSeek (source).
• Quantization, mixed-precision recipe, and the M5 Neural-Accelerator kernels: this repository's

author, via the ds4 engine.