---

base_model: turboderp/DeepSeek-V4-Flash-0731-exl3

base_model_relation: quantized

quantized_by: turboderp

library_name: gguf

license: mit

pipeline_tag: text-generation

tags:

• exl3
• ranma

---

DeepSeek-V4-Flash-0731 RANMA EXL3 GGUF

> [!WARNING]

> Requires ranma.cpp ranma_20261005 or newer (github.com/saturnsky/ranma.cpp).

> These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

> [!NOTE]

> Status: loading and generation work, but the DeepSeek-V4-Flash path in ranma.cpp has not been performance-optimized yet.

> Expect it to be slower than the Qwen3.8-Flash-Next EXL3 files. No benchmark numbers are published for this model.

What this is

This repository repackages the EXL3 quantization made by turboderp of

deepseek-ai/DeepSeek-V4-Flash-0731 into a GGUF that ranma.cpp can read.

The weights were not re-quantized. The EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container

(with the fork's EXL3 tensor types) and split into shards. All credit for the quantization goes to turboderp and the

ExLlamaV3 project. Thank you.

Source (EXL3, by turboderp): turboderp/DeepSeek-V4-Flash-0731-exl3,

branch 3.04bpw (ExLlamaV3 quantization version 1.3.0, codebook mul1, head bits 6,

calibration 250 rows x 2048).

Not included

This file holds the main model only. The source checkpoint's mtp.safetensors (7.1 GiB: the DSpark draft model, three layers plus

its prediction heads) is not carried over: the converter does not handle it yet. Speculative decoding with it is not available from

this repository. In llama.cpp a draft model is a separate GGUF, so it can be added later without changing the files here.

Files

Each quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors

(llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma. manifest.json lists the size and SHA-256 of every file.

| Folder | bpw (head) | Size | Files |

|---|---|---:|---:|

| EXL3-3.04bpw-h6 | 3.04 (6) | 110.0 GB (102.4 GiB) | 4 |

File names: DeepSeek-V4-Flash-0731-EXL3-3.04bpw-h6-0000K-of-0000N.gguf.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the

experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the

GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the examples below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only;

there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement).

The first shard is enough for -m; the other shards are found by name.

| Option | Default | Meaning | How to choose |

|---|---|---|---|

| --expert-l1-mib N | 0 (cache off) | VRAM budget in MiB of the expert cache, L1. It also decides which experts go to host memory. | VRAM - (dense weights + KV cache + compute buffers) - margin; see below. |

| --expert-l2-mib N | -1 (unlimited) | Host-RAM budget in MiB of the expert cache, L2. -1: every expert that is not in L1 lives in RAM. A finite value: experts that fit in neither L1 nor L2 stay in the GGUF file on the SSD and are read on demand (Windows + HIP). | Keep -1 if the experts outside L1 fit in the RAM you can spare. Otherwise use the RAM you can spare for the experts, minus what your other programs need; the SSD serves the rest, at some cost in speed. |

| --expert-cache-mode inclusive\|exclusive | inclusive | exclusive keeps one copy of each expert (in VRAM or in RAM); inclusive keeps a RAM copy of every VRAM resident as well. | exclusive when RAM is tight; the benchmarks use it. Windows. |

| --expert-profile-dir DIR | none | Directory where the server stores which experts each request used. Without it the placement stays a fixed random one. | Any writable directory. The placement adapts at request boundaries; about ten requests rebuild a profile. |

| --expert-prefill-swap | off | Use a separate plan while a prompt is processed. | Optional. |

| -np N | 4 in the server | Number of parallel slots. Always give -np 1 for these examples: with the default 4 slots the recurrent state and compute buffers at 256K grow by several GiB, and a finite L2 (--expert-l2-mib positive) is refused when the slot count is not fixed (finite L2 needs positive batch/parallel bounds). | -np 1 unless you serve several users. |

| --gpu-heartbeat-seconds S | 5.0 (server) | Pulses the GPU while the server is idle so that the OS does not evict VRAM. | Leave the default. |

Sizing L1 and L2

• L1. L1 = VRAM - (memory the cache does not cover) - margin. What the cache does not cover is the dense weights, the KV cache and the compute buffers;

the KV cache grows with the context (little for this model, see the example command). Reference: for Qwen3.8-Flash-Next about 8.3 to 9.5 GiB of device memory were outside the cache at context 65536; for this DeepSeek model the verification run with L1 = 16 GiB peaked at 23.7 GiB dedicated VRAM, so measure your own context size.

Keep a margin of a few GiB, more if the GPU also drives a display. The benchmark budgets were deliberately conservative; a larger L1 puts more experts in

VRAM. The L1 budget includes the cache's own tables and overhead, so it is not exactly the expert bytes.

• L2. With -1, the RAM used by the experts is about the model size minus L1 (exclusive mode); the KV cache, prompt cache and buffers come on top.

With a finite L2 the cache's host part is bounded by it. Set it to the RAM you can spare minus what your other programs use. The server refuses to load

if the host requirement is above the physical memory installed, but it does not look at how much is free at that moment.

• If L1 + L2 covers all experts, nothing is read from the file; each expert that has to come from the SSD costs speed.

The commands and values below are examples only. Pick them for your own use and system: the context length (a long roleplay needs a large -c, a short chat or a coding session can use much less and leave VRAM to L1), the split between L1 and L2, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and workload works best).

Example command (256K context)

The KV cache grows with the context, but this model keeps only a short sliding window of raw cells plus compressed caches (1 cell per 4 or 128 tokens), so it is small: about 1.8 GiB at -c 262144 (f16, estimated from the cache layout). The L1 and L2 values are the ones this conversion was verified with (L1 16 GiB exclusive, finite L2 40 GiB, the remaining experts read from the SSD, no MTP, peak private memory about 65 GiB, roughly L2 plus 24 GiB; dedicated VRAM peak 23.7 GiB at the verification context). Checked at -c 262144 -np 1 on an R9700 (32 GB): load and a short generation worked; dedicated VRAM peak 23.0 GiB, private memory peak 64.0 GiB. Adjust L1 and L2 to your machine as described above.

32 GB VRAM + 128 GB RAM:

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.04bpw-h6\DeepSeek-V4-Flash-0731-EXL3-3.04bpw-h6-00001-of-00004.gguf ^
  -c 262144 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 16384 --expert-l2-mib 40960 ^
  --expert-profile-dir .\expert-profiles

Details of every option: expert-cache.md, expert-cache-exclusive.md,

expert-cache-l2.md, host-direct-moe.md, spec-smart.md,

gpu-heartbeat.md.

License

Same as the original model: MIT, see the original repository.