---

license: other

license_name: deepseek

base_model: deepseek-ai/DeepSeek-V4-Flash

tags:

• gguf
• rocmfp
• rocmfpx
• strix-halo
• gfx1151
• amd
• deepseek-v4
• moe

pipeline_tag: text-generation

---

DeepSeek-V4-Flash — Strix Halo GGUF (ROCMFPx affine)

> # ⚠️ This will NOT run on mainline llama.cpp, LM Studio, or Ollama

>

> This is not a standard GGUF. It uses custom ROCMFPx tensor types

> (Q2_0_ROCMFP2 affine fp2, Q4_0_ROCMFP4_FAST) together with the

> DeepSeek-V4 (deepseek4) graph, and needs a ROCMFPx-aware runtime built

> for gfx1151.

>

> Upstream llama.cpp does not error — it loads the file and emits garbage.

> The Q2_K in the filename is the Hugging Face weight-class naming convention

> (~2.6 bpw); it is not the standard ggml Q2_K type.

>

> The required runtime is not publicly released yet. It is Otheru's

> from-scratch C engine (ember), so these weights currently cannot be run

> outside our own builds. They are published for reference and reproducibility.

A full ROCMFPx quantization of DeepSeek-V4-Flash tuned for the AMD Strix Halo APU

(Ryzen AI Max, gfx1151, 128 GB unified memory). Every MoE expert — **including the

sensitive down-projection — is carried at 2.5 bpw* using a new affine* fp2 format,

giving a coherent model at 85.3 GiB / 2.58 bpw that leaves real headroom on a 128 GB

machine (no swap thrash under concurrent load).

Benchmarks (Strix Halo, gfx1151, DSpark spec-decode)

| metric | this model |

|---|---|

| sparse prefill | 248–253 tok/s |

| decode (clean-gen) | 32 tok/s (accept-rate dependent; ~28–29 avg on mixed prompts) |

| size on disk | 85.3 GiB (2.58 bpw) |

| RAM resident | ~89 GB |

Meets/exceeds the reference Strix-Halo targets of 245 tok/s prefill and 32 tok/s decode.

The affine fp2 format

Two bits gives exactly four representable values, so none of them may be wasted.

The stock ROCMFP fp2 is symmetric: value = code · scale, a fixed 4-entry codebook

with one scale per half-block. That spends its four levels on the assumption that each

block of weights is centred on zero and roughly balanced. Real blocks usually are not —

a given 32-weight block might live almost entirely in, say, [-0.02, +0.31] — so levels

land where there is little mass and the part that matters is left with roughly one bit of

resolution. On the MoE down-projection that was coarse enough to produce garbage output.

This build uses an affine (asymmetric) scheme instead:

value = code · scale − offset        codes ∈ {0, 1, 2, 3}

Affine is the standard term for a linear map plus a translation (y = ax + b). The

offset is that second degree of freedom, and it lets the four levels be fitted to the

block's actual range instead of an assumed zero-centred one:

code 0  →  −offset           = min    ⇒   offset = −min
code 3  →  3·scale − offset  = max    ⇒   scale  = (max − min) / 3

Every level now does work. Measured quantization error drops to NRMSE 0.292, against

0.287 for Q2_K at a comparable rate — and 0.376 for the symmetric codebook it replaces.

It costs nothing

Per 32-weight block:

| field | bytes |

|---|---|

| packed 2-bit codes (32 × 2 bits) | 8 |

| scale (UE4M3) | 1 |

| offset (UE4M3) | 1 |

| total | 10 bytes = 2.5 bpw |

Identical to the symmetric layout it replaces. The old format already spent two bytes on

two per-half-block scales, so the second one simply becomes the offset: same block size,

same bit rate, same kernel shape, one extra degree of freedom for free. That is why the

accuracy gain does not cost throughput.

Because scale and offset are unsigned UE4M3, offset ≥ 0; a block whose minimum

is positive gets offset = 0 and spans [0, 3·scale].

This is the same "scale and min" idea as llama.cpp's Q2_K, expressed in the ROCMFPx

block layout. Both the GPU kernels and the CPU reference path implement exactly the

formula above.

References

• Jacob et al., *Quantization and Training of Neural Networks for Efficient

Integer-Arithmetic-Only Inference* (2017) — the canonical affine formulation

r = S·(q − Z), i.e. scale plus zero-point:

arXiv:1712.05877

• Krishnamoorthi, *Quantizing Deep Convolutional Networks for Efficient Inference: A

Whitepaper* (2018) — symmetric vs. asymmetric, per-tensor vs. per-channel:

arXiv:1806.08342

• Nagel et al., A White Paper on Neural Network Quantization (2021) — modern treatment

of affine quantization and range estimation:

arXiv:2106.08295

• ikawrakow, k-quants — the Q2_K…Q6_K block scale-and-min scheme this mirrors:

ggml-org/llama.cpp#1684

Quant recipe (per tensor)

• ffn_{gate,up,down}_exps → Q2_0_ROCMFP2 (affine fp2, 2.5 bpw)
• shared experts, attention, dense, indexer, output → Q4_0_ROCMFP4_FAST
• attn_output_b → Q8_0 · token_embd → Q6_K · norms/bias/sinks → F32
• blk.0 / blk.43 routed experts → MXFP4 (preserved)
• imatrix-calibrated (routed-MoE aligned)

> Filename note: the Q2_K tag follows the HF GGUF naming convention as the

> ~2.6 bpw weight class (this build is 2.58 bpw and the affine experts implement

> Q2_K's scale+min scheme). It is not the standard Q2_K ggml type and will

> not load in upstream llama.cpp — see Runtime. File: DeepSeek-V4-Flash-ROCMFPx.Q2_K.gguf.

Runtime

This model uses custom ROCMFPx tensor types and the DeepSeek-V4 (deepseek4) graph.

It requires a ROCMFPx-aware runtime built for gfx1151 (the affine fp2 mmvq/mmq kernels).

It will not load in upstream llama.cpp.

Provenance

Converted from the base deepseek-ai/DeepSeek-V4-Flash FP8 release → BF16 (modular

DeepSeek-V4 converter) → ROCMFPx affine quantize. Not abliterated / not fine-tuned.

License

Inherits the DeepSeek-V4 model license. See the base model card.