---

base_model: turboderp/GLM-5.3-Flash-exl3

base_model_relation: quantized

quantized_by: turboderp

library_name: gguf

license: mit

pipeline_tag: text-generation

tags:

• exl3
• ranma

---

GLM-5.3-Flash RANMA EXL3 GGUF

> [!WARNING]

> Requires ranma.cpp ranma_20261007 or newer (github.com/saturnsky/ranma.cpp).

> These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

> [!NOTE]

> No benchmark numbers are published for this model.

What this is

This repository repackages the EXL3 quantizations made by turboderp of

zai-org/GLM-5.3-Flash into GGUF files that ranma.cpp can read.

The weights were not re-quantized. The EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container

(with the fork's EXL3 tensor types) and split into shards. The fused KDA qkv_proj (and its conv1d) of the checkpoint is stored as the separate q, k and v

tensors that the GGUF graph reads, cut at 128-row boundaries without changing the trellis bytes. All credit for the quantization goes to turboderp and the

ExLlamaV3 project. Thank you.

Sources (EXL3, by turboderp), one branch per quantization. Each source branch maps to one folder in this repository:

| Source branch (turboderp) | Folder in this repo |

|---|---|

| 2.05bpw | EXL3-2.05bpw-h5 |

| 3.05bpw | EXL3-3.05bpw-h6 |

Both were produced with ExLlamaV3 quantization version 1.4.4 (codebook mul1, calibration 250 rows x 2048). h5/h6 are the head bits

(output layer).

The multi-token-prediction (MTP) layer of the checkpoint is included in the same file; see Running.

Not included

| Part | Size in the source | Status |

|---|---|---|

| Vision encoder | 0.6 GB | Not converted: the converter does not handle quantized vision towers yet. Image and video input is not available with these files; no mmproj file is provided. |

In llama.cpp the vision encoder is a separate mmproj file, so it can be added later without changing the files here.

Files

Each quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors

(llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma; the rest are found automatically.

manifest.json lists the size and SHA-256 of every file.

| Folder | bpw (head) | Size | Files |

|---|---|---:|---:|

| EXL3-2.05bpw-h5 | 2.05 (5) | 84.7 GB (78.9 GiB) | 3 |

| EXL3-3.05bpw-h6 | 3.05 (6) | 124.7 GB (116.1 GiB) | 4 |

File names: GLM-5.3-Flash-EXL3-<bpw>-<head>-0000K-of-0000N.gguf.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the

experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the

GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the example below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only;

there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement).

The first shard is enough for -m; the other shards are found by name.

| Option | Default | Meaning | How to choose |

|---|---|---|---|

| --expert-l1-mib N | 0 (cache off) | VRAM budget in MiB of the expert cache, L1. It also decides which experts go to host memory. | VRAM - (dense weights + KV cache + compute buffers) - margin; see below. |

| --expert-l2-mib N | -1 (unlimited) | Host-RAM budget in MiB of the expert cache, L2. -1: every expert that is not in L1 lives in RAM. A finite value: experts that fit in neither L1 nor L2 stay in the GGUF file on the SSD and are read on demand (Windows + HIP). | The RAM you can spare for the experts, minus what your other programs need; the SSD serves the rest, at some cost in speed. |

| --expert-cache-mode inclusive\|exclusive | inclusive | exclusive keeps one copy of each expert (in VRAM or in RAM); inclusive keeps a RAM copy of every VRAM resident as well. | exclusive. Windows. |

| --expert-profile-dir DIR | none | Directory where the server stores which experts each request used. Without it the placement stays a fixed random one. | Any writable directory. The placement adapts at request boundaries; about ten requests rebuild a profile. |

| --spec-type draft-mtp | none | Speculative decoding with the MTP layer in this file. No -md: the MTP layer is read from the same GGUF, and its experts share the expert cache with the main model. | Optional. It needs about 1.5 GiB more VRAM, so lower L1 by that much. |

| --spec-smart (default on), --spec-smart-store PATH | on, none | Chooses the draft length at every step from measured times. The store file keeps the estimates across restarts. | With MTP: give a store path so a restart is not cold. |

| -np N | 4 in the server | Number of parallel slots. Always give -np 1 for this example: a finite L2 (--expert-l2-mib positive) is refused when the slot count is not fixed (finite L2 needs positive batch/parallel bounds). | -np 1 unless you serve several users. |

| --gpu-heartbeat-seconds S | 5.0 (server) | Pulses the GPU while the server is idle so that the OS does not evict VRAM. | Leave the default. |

Sizing L1 and L2

• L1. L1 = VRAM - (memory the cache does not cover) - margin. What the cache does not cover is the dense weights (about 5.5 GiB in VRAM for 3.05 bpw),

the KV cache and the compute buffers. Most layers of this model are linear-attention (KDA) layers with a fixed-size state, so the KV cache is small:

about 1.2 GiB at -c 65536 (f16). With L1 = 20480 at -c 65536 the dedicated VRAM peaked at about 28.6 GiB on a 32 GB R9700.

Keep a margin of a few GiB, more if the GPU also drives a display. The L1 budget includes the cache's own tables and overhead, so it is not exactly the

expert bytes.

• L2. The routed experts take 71.3 GiB (2.05 bpw) and 106.7 GiB (3.05 bpw). In exclusive mode L1 and L2 hold different experts, so L1 + L2 at or

above that size reads nothing from the SSD. The physical RAM in use is about L2 plus a few GiB; Windows reports a higher private memory, because it also

counts the VRAM and the mapped host memory. The server refuses to load if the host requirement is above the physical memory installed, but it does not

look at how much is free at that moment.

The command and values below are an example only. Pick them for your own use and system: the context length, the split between L1 and L2, whether to

use MTP, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and

workload works best).

Example command (64K context)

32 GB VRAM + 128 GB RAM:

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.05bpw-h6\GLM-5.3-Flash-EXL3-3.05bpw-h6-00001-of-00004.gguf ^
  -c 65536 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 20480 --expert-l2-mib 92160 ^
  --expert-profile-dir .\expert-profiles

For 2.05 bpw use EXL3-2.05bpw-h5\GLM-5.3-Flash-EXL3-2.05bpw-h5-00001-of-00003.gguf; its experts fit in L1 + L2 entirely, so a smaller L2 is enough.

For MTP add --spec-type draft-mtp --spec-smart-store .\smart-store.json and lower L1 to --expert-l1-mib 18944.

The chat template of the original model is in the file. Its prompts start with [gMASK]<sop>; use the chat endpoint (or add that prefix yourself for

raw completion).

Details of every option: expert-cache.md, expert-cache-exclusive.md,

expert-cache-l2.md, expert-cache-joint.md,

host-direct-moe.md, spec-smart.md,

gpu-heartbeat.md, exl3.md.

License

Same as the original model: MIT, see the original repository.