---

base_model: zai-org/GLM-5.2

base_model_relation: quantized

tags:

• gguf
• glm
• moe
• ds4
• ssd-streaming

---

GLM-5.2 GGUF for ds4 (SSD streaming, CUDA)

This is a mirror of the official ds4 GGUF of GLM-5.2

(743B MoE) built by antirez and published at

antirez/GLM-5.2-GGUF (bit-identical file, same

sha256). Credit for the quantization is his; this repo re-documents it with the full

per-tensor recipe below and pairs it with the CUDA/SSD-streaming usage notes. It is the file

used by the ds4 inference engine, specifically the

glm-local branch which adds the CUDA port,

SSD expert streaming optimizations, and the first MTP speculative-decoding implementation for

GLM 5.2 on any backend.

The design target is machines that cannot hold the model in RAM at all: routed experts are

read from disk per token while the ~20 GiB of attention/shared weights stay resident. On an

RTX 4060 Ti 16GB + 30GB RAM + a Gen4 NVMe it decodes at ~0.4 tokens/s and climbs with disk

bandwidth. Yes, that is slow. It is also a 743B model answering on hardware worth less than

a mid-range gaming rig.

Files

| File | Size | Routed experts (ffn_{gate,up,down}_exps) | Everything else |

|------|------|------------------------------------------|-----------------|

| GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf | 196.6 GiB | IQ2_XXS (layers 3-77, 225 tensors, 177 GiB); blk.78 MTP layer at Q2_K | Q8_0 attention/shared-expert/embeddings/output (872 tensors) + F32 norms (709 tensors), 19.6 GiB |

The MTP draft head (blk.78: full GLM layer + nextn eh_proj/enorm/hnorm/shared_head_norm) is

included in the main file. No separate draft gguf is needed: pass the same file to --mtp.

Quantization recipe

The filename is the spec. In detail:

| Tensor class | Quant | Notes |

|---|---|---|

| blk.*.ffn_{gate,up,down}_exps (layers 3-77) | IQ2_XXS | routed experts, uniform on purpose: the streaming expert cache uses fixed-size slabs and the dp4a decode kernels read IQ2_XXS directly |

| blk.78.ffn_{gate,up,down}_exps | Q2_K | the MTP draft layer's experts; never runs in the main decode loop, only feeds speculation |

| blk.*.ffn_{gate,up,down}_shexp | Q8_0 | shared experts |

| blk.{0,1,2}.ffn_{gate,up,down} | Q8_0 | leading dense layers |

| blk.*.attn_q_a, attn_q_b, attn_kv_a_mqa, attn_k_b, attn_v_b, attn_output | Q8_0 | all MLA attention projections |

| blk.*.indexer.attn_q_b, indexer.attn_k | Q8_0 | DSA sparse-indexer projections |

| blk.*.indexer.proj, indexer.k_norm(+bias) | F32 | indexer scoring head |

| blk..ffn_gate_inp (router), blk..exp_probs_b (router bias) | F32 | learned router, kept exact |

| blk.78.nextn.eh_proj | Q8_0 | MTP embed/hidden fusion |

| blk.78.nextn.{enorm,hnorm,shared_head_norm} | F32 | MTP glue norms |

| token_embd.weight, output.weight | Q8_0 | embeddings and output head |

| all *_norm.weight | F32 | |

The motivation behind the asymmetry: the routed experts are the majority of the parameter

count but each individual expert handles only a fraction of tokens, so aggressive

quantization on them costs less in average quality than the same treatment of the router,

projections, or shared experts. Keeping the decision-making components at Q8_0 preserves

model behavior; crushing the experts buys the size.

SSD streaming adds a second reason: the Q8_0/F32 set (~20 GiB) is resident, so its bytes

are paid once in RAM, while the experts are read from disk again and again. Quantizing the

experts harder is a per-token bandwidth win; quantizing the resident set harder would save

only idle memory. The asymmetry follows the traffic, not just the parameter count.

Usage

Needs the glm-local branch of ds4 (CUDA, sm_89 tested) and a fast NVMe. Expert reads are

O_DIRECT through io_uring; host RAM is used for a popularity (LFU) expert cache, so give it

whatever you can spare with DS4_CUDA_HOST_EXPERT_CACHE_GB.

git clone -b glm-local https://github.com/giannisanni/neutronstar
cd ds4 && make cuda CUDA_ARCH=sm_89
DS4_GLM_CUDA_UNSAFE=1 DS4_CUDA_HOST_EXPERT_CACHE_GB=7 DS4_CUDA_PARALLEL_FETCH_THREADS=16 \
./ds4 -m GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --cuda --ssd-streaming --ssd-streaming-cache-experts 64 \
  --ctx 4096 --tokens 400 --nothink -p "Tell me something surprising about Suriname."

Interactive chat: drop -p. MTP probe telemetry: add --mtp <same file> with DS4_MTP_PROBE=1

and DS4_MTP_STREAMING_UNSAFE=1.

Measured on RTX 4060 Ti 16GB / 30GB DDR5 / Gen4 x4 NVMe: prefill ~0.35 t/s, generation

~0.40 t/s with a 7 GiB host expert cache (~30% hit rate: the hottest 4% of experts serve

30% of lookups). The engine runs at ~89% of the PCIe link ceiling; a faster disk moves the

number almost linearly.

sha256: a49de64c5020432bdae23de36a423a9660a5621bc0db8d12b66bd8814b07fea0

License

Inherits the upstream GLM-5.2 model license (zai-org). The quantization recipe and this

card: MIT.