---

license: mit

base_model: zai-org/GLM-5.2

pipeline_tag: text-generation

tags:

- moe

- reap

- pruning

- router-kd

- gguf

- llama.cpp

- glm

- glm-5.2

---

GLM-5.2-REAP-504B-GGUF — BF16 + dynamic 2/3/4-bit

> GGUF builds of the 34%-expert-pruned, Router-KD-recovered GLM-5.2-504B — for llama.cpp

> (CPU / Metal / CUDA). Includes the first working GGUF of a GLM-5.2 DSA model, made possible by

> a custom patch for its shared sparse-attention indexer.

The full-precision NVFP4 model lives at 0xSero/GLM-5.2-504B — start there for the model story, eval, and architecture. This card is about the GGUF conversion specifically.

---

🙏 Sponsor

Built on 8× NVIDIA B200 sponsored by Lambda. Thank you. 🙏

---

Files

| file | bits | size | notes |

|---|---|---|---|

| GLM-5.2-REAP-504B-BF16 | 16 | ~933 GB | full precision — for fine-tuning / re-quantizing (23 shards) |

| GLM-5.2-REAP-504B-Q4_K_XL | ~4.5 | ~325 GB | recommended — dynamic, highest usable quality (8 shards) |

| GLM-5.2-REAP-504B-Q3_K_XL | ~3.5 | ~259 GB | dynamic, strong quality/size (6 shards) |

| GLM-5.2-REAP-504B-Q2_K_XL | ~2.7 | ~111 GB | dynamic, smallest (5 shards) |

Files > 45 GB are split (...-00001-of-000NN.gguf); point llama.cpp at the first shard and it

loads the rest automatically.

What "dynamic" means here

These are not flat K-quants. Each uses per-tensor precision: the parts that hurt most under

quantization are kept high while the bulk (routed experts) is pushed low:

• token-embedding → Q6_K, output head → Q8_0/Q6_K
• attention (MLA q/kv) → Q6_K / Q5_K / Q4_K (by tier)
• shared expert → Q8_0 / Q6_K
• routed-expert down-projection kept above gate/up (it's the sensitive one)

> Not yet imatrix-calibrated. A true importance-matrix pass needs the model running under

> llama.cpp, and our llama.cpp build is CPU-only while the GPUs were committed elsewhere. The

> per-tensor recipe still puts these well above naive quants; imatrix-calibrated builds may follow.

The hard part — making GLM-5.2 loadable in llama.cpp

GLM-5.2 uses *DSA (DeepSeek-style sparse attention) with a shared indexer*: only ~1 layer in 4 is

a "full" indexer layer; the other 57 of 78 layers share it and carry **no indexer weights of

their own (index_topk_freq: 4). Stock llama.cpp demands an indexer tensor on every** layer, so

the converted GGUF failed to load:

missing tensor 'blk.3.indexer.k_norm.weight'

The fix (applied to these files): we **filled each of the 57 shared layers' indexer tensors

(285 tensors total) by duplicating them from the nearest preceding full layer**, making the GGUF

loadable.

The MTP / next-token-prediction layer (blk.78) is preserved — kept at Q6_K in every quant

(the stock quantizer rejects the nextn layer's experts, so we pin them with explicit per-tensor

types). This keeps self-speculative decoding available in llama.cpp for this architecture.

Verified: the patched model loads and generates coherent text end-to-end in llama-cli.

> Caveat (honest): the sharing is approximated — the duplicated weights are recomputed per

> layer rather than the index being reused exactly as the original does. Output is coherent in

> testing, but this is not a bit-exact reproduction of the reference attention. Prefer Q4_K_XL or

> BF16 where fidelity matters most; the low-bit quants were not individually loop-tested.

Run it

# loads all shards from the first one
llama-cli    -m GLM-5.2-REAP-504B-Q4_K_XL-00001-of-00008.gguf -p "Hello"
llama-server -m GLM-5.2-REAP-504B-Q4_K_XL-00001-of-00008.gguf --host 0.0.0.0 --port 8080

Tips (measured at n=2000): a sampler guardrail is the single best free anti-loop knob —

--min-p 0.05 --repeat-penalty 1.05 → loop 7.2%→4.9%; --repeat-penalty 1.10 → 2.3%, i.e. it

fully recovers the pruning-induced looping (raw 7.2%; unpruned teacher 3.6%). Use 1.05 by default,

1.10 if you see loops (higher penalty can slightly over-suppress legitimate repetition in code). A

"Be concise…" system prompt separately halves output length (it shortens, doesn't de-loop).

How the underlying model was made (short version)

REAP expert-pruning (gate × ‖expert_output‖) → keep 168/256 experts/layer (~504B, 34% pruned) →

gate-only Router-KD to the unpruned teacher (train 0.016% of params). On n=2000 held-out real

prompts it loops at 7.2% vs the unpruned teacher's 3.6% — pruning roughly doubles the loop rate

(gate-KD recovers routing but not full termination; a knowledge-recovery LoRA is in progress to close

the gap). Full details + honest accounting on the NVFP4 model card.

---

GGUF conversion + custom DSA-indexer patch. Compute sponsored by Lambda — thank you. 🙏