---

license: mit

library_name: gguf

base_model: puwaer/DeepSeek-V4-Flash-0731-ream-200b

base_model_relation: quantized

pipeline_tag: text-generation

tags:

• moe
• expert-merging
• ream
• deepseek_v4
• compressed
• gguf
• llama.cpp

---

puwaer/DeepSeek-V4-Flash-0731-ream-200b-gguf

GGUF builds of puwaer/DeepSeek-V4-Flash-0731-ream-200b

for llama.cpp — 178 routed experts per layer (down from the base model's 256)

via router-weighted expert activation merging (REAM). See that repo's card

for the compression recipe and full benchmark discussion; this one covers what

is specific to the GGUF files: what each one is, how it was quantized, and how

to serve it.

Produced with moe-compress and

llama.cpp.

Files

| File | Type | Size | imatrix |

|---|---|---|---|

| *-MXFP4_MOE.gguf | baseline | 103.85 GiB | not used — see below |

| *-Q3_K_M.gguf | quantized | 86.55 GiB | required |

| *-IQ3_XXS.gguf | quantized | 81.60 GiB | required |

| *-Q2_K.gguf | quantized | 75.82 GiB | required |

| *-IQ1_M.gguf | quantized | 56.36 GiB | required |

MXFP4_MOE is not a quantization, it is the baseline. DeepSeek ships this

model's routed experts already in MXFP4 (expert_dtype: fp4); llama.cpp's

converter repacks those values into ggml's MXFP4 block layout without changing

a single one (conversion/deepseek.py::_pack_mxfp4_blocks, 17 bytes per 32

values either way). So this file is numerically identical to the safetensors

checkpoint linked above — same weights, different container.

Every rung below it is a requantization of already-4-bit data, not a

quantization from bf16 (--allow-requantize is required), and costs somewhat

more quality than the same rung would starting from a full-precision

checkpoint. That is true of every public GGUF of this model, not a property of

this pipeline.

Attention, the indexer and the shared experts stay Q8_0 at every rung; the

entire size budget below MXFP4_MOE goes to the routed experts

(ffn_{gate,down,up}_exps), which are ~90% of the checkpoint.

Quantization recipe

Built with a commit-pinned llama.cpp

(42e98813e, the

first commit with DeepSeek-V4 support) on Hopper (sm_90):

python3 convert_hf_to_gguf.py DeepSeek-V4-Flash-0731-ream-200b --outtype bf16 \
    --outfile DeepSeek-V4-Flash-0731-ream-200b-MXFP4_MOE.gguf

llama-imatrix -m DeepSeek-V4-Flash-0731-ream-200b-MXFP4_MOE.gguf \
    -f imatrix-calibration.txt -c 512 -ngl 99 \
    --output-format gguf -o imatrix.gguf

llama-quantize --allow-requantize --imatrix imatrix.gguf \
    --tensor-type-file recipes/q3km.txt \
    DeepSeek-V4-Flash-0731-ream-200b-MXFP4_MOE.gguf \
    DeepSeek-V4-Flash-0731-ream-200b-Q3_K_M.gguf Q3_K_M

(repeated per rung, swapping the recipe file and ftype: iq3xxs.txt/IQ3_XXS,

q2k.txt/Q2_K, iq1m.txt/IQ1_M)

**The imatrix is computed against this checkpoint's own 178 experts, not

borrowed.** The only public DeepSeek-V4-Flash-0731 imatrix

(tarruda/DeepSeek-V4-Flash-0731-GGUF)

is sized for the base model's 256 experts; llama-quantize checks

imatrix_size == ne[0]*ne[2] and refuses a mismatch outright. The calibration

corpus is that repo's own imatrix-calibration.txt, so the numbers stay

comparable to a known-good reference. Because these are REAM's merged experts,

this imatrix also carries their own statistics rather than those of whichever

original expert happened to become each survivor's centroid.

Serving

llama-server -m DeepSeek-V4-Flash-0731-ream-200b-Q3_K_M.gguf \
    -ngl 99 --ctx-size 4096

The chat template is embedded in the GGUF (the same chat_template.jinja as

the safetensors checkpoint — thinking on by default, see below), so no

--chat-template flag is needed. Pass --reasoning-format deepseek to split

the <think>...</think> block into message.reasoning_content instead of

leaving it inline in message.content.

Full GPU offload (-ngl 99) is what this pipeline verified against.

Batched CUDA prefill of a partially-offloaded DeepSeek-V4 graph has been

observed to fail (GGML_ASSERT in mul_mat_id) on checkpoints where a

token's routing row names the same expert twice; this checkpoint has that

fixed, but low -ngl was not re-tested after the fix.

This checkpoint is 103.85 GiB even at the smallest rung above — it does not

fit one 96 GB card. -ngl below 99 with the rest on host RAM, or a multi-node

rpc-server split, are both untested here.

Benchmarks

The scores below are the source checkpoint's, measured with SGLang on the

safetensors weights — see

puwaer/DeepSeek-V4-Flash-0731-ream-200b

for the full discussion. They carry over exactly to MXFP4_MOE, which is

bit-identical to that checkpoint. The quantized rungs (Q3_K_M and below) have

not been benchmarked separately — expect some further accuracy cost, unmeasured

here.

| Model | Experts | Size | GSM8K | MATH-500 | HumanEval+ | MBPP+ | mean |

|---|---|---|---|---|---|---|---|

| base 284b | 256 | 156 GiB | 0.9484 | 0.7060 | 0.8720 | 0.7407 | 0.8168 |

| REAP 200b | 178 | 104 GiB | 0.9401 | 0.6880 | 0.8720 | 0.7407 | 0.8102 |

| REAM 200b | 178 | 104 GiB | 0.8620 | 0.6080 | 0.8841 | 0.7698 | 0.7810 |

| REAP 150b | 132 | 79 GiB | 0.9295 | 0.7140 | 0.8963 | 0.7593 | 0.8248 |

| REAM 150b | 132 | 79 GiB | 0.6922 | 0.5020 | 0.8537 | 0.7328 | 0.6952 |

Difference from the base model, in points:

| GSM8K | MATH-500 | HumanEval+ | MBPP+ | mean |

|---|---|---|---|---|

| -8.64 | -9.80 | +1.22 | +2.91 | -3.58 |

Metrics: GSM8K exact_match,strict-match, MATH-500 math_verify,none,

HumanEval+/MBPP+ pass@1_plus. All greedy (n=1), 4096-token context,

enable_thinking=false.

What changed relative to the base model

| | base | this model |

|---|---|---|

| Routed experts per layer | 256 | 178 |

| Decoder layers | 43 | 43 |

| Experts per token | 6 | 6 |

| Checkpoint size (safetensors) | 156 GiB | 104 GiB |

| MTP modules (mtp.0/1/2) | present (4705 tensors) | absent |

| chat_template | not shipped | shipped, embedded in the GGUF |

| encoding/encoding_dsv4.py | present | not part of a GGUF |

One difference deserves to be read before you deploy this:

• The multi-token-prediction modules are gone. The base checkpoint carries

mtp.0, mtp.1 and mtp.2; this one carries none of them. **MTP-based

speculative decoding is therefore unavailable.** Engines that look for those

weights will fall back to ordinary decoding. Nothing else references them, so

standard generation is unaffected.

Everything else about the prompt and generation defaults is the base model's.

Thinking

Thinking is on by default, just like the base model.

# Python, against the safetensors checkpoint
tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=False)

# llama-server / SGLang API
{"messages": [...], "chat_template_kwargs": {"enable_thinking": false}}

• Reasoning effort: Set reasoning_effort to "low" (default), "high", or "max" (applies only in thinking mode).

Practical Notes

1. Token Budget: Thinking consumes tokens before the actual answer starts. Ensure your context/output budget is high enough to prevent mid-reasoning truncation.
2. Default Sampling: The safetensors checkpoint samples by default (do_sample, temperature, top_p) rather than using greedy decoding; set your own sampling params when serving the GGUF, since generation_config.json is not embedded in it.
3. Output Format: Reasoning appears inline in message.content, ending with </think>, unless split out — with llama.cpp, --reasoning-format deepseek moves it to message.reasoning_content.

Choosing between REAP and REAM

On this model REAP wins outright, and by a margin that widens as more is

removed. Points against the base model, given as 178 experts / 132 experts:

| | GSM8K | MATH-500 | HumanEval+ | MBPP+ | mean |

| --- | --- | --- | --- | --- | --- |

| REAP — prune low-saliency experts | −0.83 / −1.90 | −1.80 / +0.80 | ±0.00 / +2.44 | ±0.00 / +1.85 | −0.66 / +0.80 |

| REAM — merge them into survivors | −8.64 / −25.63 | −9.80 / −20.40 | +1.22 / −1.83 | +2.91 / −0.79 | −3.58 / −12.16 |

REAP at 178 experts returns the base model's pass@1 exactly on both code

benchmarks — 542 problems, not one of them different — for a third off the

checkpoint.

Note also how differently the two scale. Going from 178 experts to 132 costs

REAP 1.1 more points of GSM8K; it costs REAM 17.0.

Citation

The methods:

• REAP — Router-weighted Expert Activation Pruning. Lasby et al., 2025.

arXiv:2510.13999

• REAM — Router-weighted Expert Activation Merging. Jha et al., 2026.

arXiv:2604.04356

The compression implementation: https://github.com/puwaer/moe-expert-compress

The GGUF conversion and quantization: https://github.com/ggml-org/llama.cpp

The base model: deepseek-ai/DeepSeek-V4-Flash-0731

License

MIT, following the base model. The compression code is MIT; its numerical core

is ported from the official REAM reference implementation (Copyright (c) 2026

Samsung Electronics Co., Ltd.) with attribution headers retained. llama.cpp is

MIT-licensed and imposes no further terms on its output.