---

base_model: 0xSero/GLM-5.3-Flash-EXL3-Spark

base_model_relation: quantized

library_name: gguf

license: mit

pipeline_tag: text-generation

tags:

• exl3
• ranma

---

GLM-5.3-Flash-EXL3-Spark RANMA EXL3 GGUF

> [!WARNING]

> Requires ranma.cpp ranma_20261007 or newer (github.com/saturnsky/ranma.cpp).

> These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

> [!NOTE]

> No benchmark numbers are published for this model.

What this is

This repository repackages the EXL3 weights of 0xSero/GLM-5.3-Flash-EXL3-Spark into GGUF files that ranma.cpp can read. The source artifact takes the 2.05 bpw quantization made by turboderp and raises twelve of the forty-five MoE layers (layers 3, 32, 33 and 36–44, all 288 experts of each) to 3.05 bpw, for an average of about 2.31 bpw; every other layer, the attention and embedding weights, the tokenizer and the MTP layer are byte-for-byte the base. The weights in this repository are not re-quantized either: the EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container (with the fork's EXL3 tensor types) and split into shards. All credit for the quantization goes to turboderp for the base and to 0xSero for the twelve upgraded layers, and to the ExLlamaV3 project. Thank you.

| Source (by 0xSero, from the 2.05 bpw base by turboderp) | Folder in this repo |

|---|---|

| 0xSero/GLM-5.3-Flash-EXL3-Spark | EXL3-Spark |

Produced with ExLlamaV3 quantization version 1.4.4 (codebook mul1, calibration 250 rows x 2048). h5 is the head bits

(output layer).

The multi-token-prediction (MTP) layer of the checkpoint is included in the same file; see Running.

Not included

| Part | Status |

|---|---|

| Vision encoder | Not converted: the converter does not handle quantized vision towers yet. Image and video input is not available with these files; no mmproj file is provided. |

In llama.cpp the vision encoder is a separate mmproj file, so it can be added later without changing the files here.

Files

The quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors

(llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma; the rest are found automatically.

manifest.json lists the size and SHA-256 of every file.

| Folder | bpw (head) | Size | Files |

|---|---|---:|---:|

| EXL3-Spark | ~2.31 avg (5) | 95.6 GB (89.0 GiB) | 3 |

File names: GLM-5.3-Flash-EXL3-Spark-0000K-of-00003.gguf.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the

experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the

GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the example below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only;

there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement).

The first shard is enough for -m; the other shards are found by name.

| Option | Default | Meaning |

|---|---|---|

| --expert-l1-mib N | 0 (cache off) | VRAM budget in MiB of the expert cache, L1. It also decides which experts go to host memory. |

| --expert-l2-mib N | -1 (unlimited) | Host-RAM budget in MiB of the expert cache, L2. -1: every expert that is not in L1 lives in RAM. A finite value: experts that fit in neither L1 nor L2 stay in the GGUF file on the SSD and are read on demand. |

| --expert-cache-mode inclusive\|exclusive | inclusive | exclusive keeps one copy of each expert (in VRAM or in RAM); inclusive keeps a RAM copy of every VRAM resident as well. |

| --expert-profile-dir DIR | none | Directory where the server stores which experts each request used. Without it the placement stays a fixed random one. |

| --spec-type draft-mtp | none | Speculative decoding with the MTP layer in this file. No -md: the MTP layer is read from the same GGUF, and its experts share the expert cache with the main model. It needs about 1.5 GiB more VRAM, so lower L1 by that much. |

| --spec-smart (default on), --spec-smart-store PATH | on, none | Chooses the draft length at every step from measured times. The store file keeps the estimates across restarts. |

| -np N | 4 in the server | Number of parallel slots. A finite L2 (--expert-l2-mib positive) is refused when the slot count is not fixed. |

The command and values below are an example only. Pick them for your own use and system: the context length, the split between L1 and L2, whether to

use MTP, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and

workload works best).

export GGML_CUDA_HOST_DIRECT=1
export GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-Spark/GLM-5.3-Flash-EXL3-Spark-00001-of-00003.gguf \
  -c 65536 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none \
  --expert-cache-mode exclusive --expert-l1-mib 20480 --expert-l2-mib 92160 \
  --expert-profile-dir ./expert-profiles

The routed experts of this build take 83.1 GiB in total, so an L1 + L2 at or above that reads nothing from the SSD.

Details of every option: expert-cache.md, expert-cache-exclusive.md,

expert-cache-l2.md, expert-cache-joint.md,

host-direct-moe.md, spec-smart.md,

gpu-heartbeat.md, exl3.md.

License

Same as the original model: MIT, see the original repository.