---

license: mit

base_model: zai-org/GLM-5.2

base_model_relation: quantized

pipeline_tag: text-generation

tags:

- gguf

- glm-dsa

- moe

- cpu-inference

- imatrix

- speculative-decoding

- llama.cpp

- ik_llama.cpp

---

GLM-5.2 — "e-waste edition" GGUF

Imatrix GGUF quantizations of GLM-5.2 (745B total / ~40B active, glm-dsa arch, DeepSeek-style

MoE with 256 routed + 1 shared expert, 8 active/token, MLA attention) tuned for **MoE inference on

older / "e-waste" hardware** — pre-AVX-512 CPUs and previous-gen datacenter GPUs (MI100 / gfx908).

The guiding choice across every build (Q4_K / Q3_K / Q2_K) is **K-quants for the routed experts,

not codebook i-quants (IQ2/IQ3)**: K-quant dequantization is dramatically faster on pre-AVX-512 CPUs and on

gfx908, where expert dequant — not bandwidth — is the decode bottleneck. An i-quant that looks

smaller on paper is slower to run on this class of hardware. (The experimental Q2_K_XL build keeps the

same principle — experts stay Q2_K — except for its 13 coldest, rarely-fired layers; see below.)

Available quantizations

| Variant | Size | Routed experts | Non-experts | Target |

|---|---|---|---|---|

| GLM-5.2-Q4_K_XL | 403.8 GiB | Q4_K | Q8_0 | Highest quality; big-RAM CPU-expert boxes (≈512 GB). Too big for 10×32 GB 0-spill |

| GLM-5.2-Q3_K_XL | 312.9 GiB | Q3_K | Q8_0 | Max quality on GPU; experts spill to CPU on <11×32GB |

| GLM-5.2-Q3_K_M ⭐ recommended | 295.7 GiB | Q3_K + Q2_K (cold layers) | Q6_K | Fits 0-spill on 10×32 GB (320 GB HBM) |

| GLM-5.2-Q2_K_XL | 226.9 GiB | Q2_K + IQ2_XXS (13 coldest layers) | Q4_K | Fits 0-spill on 8×32 GB (256 GB HBM); fastest decode, lowest quality |

All are imatrix-quantized. Q3_K_M is the recommended build: 17 GiB smaller than Q3_K_XL at

near-identical quality (+0.6% perplexity), exactly enough to keep the entire model resident across

ten 32 GB MI100s with zero CPU spill — eliminating the expert-dequant decode bottleneck the whole

project exists to fight.

Q2_K_XL trades quality for a much smaller footprint: at **226.9 GiB it fits 0-spill across eight

32 GB cards** (256 GB HBM) where Q3_K_M needs ten, and because it moves fewer bytes per token it

actually decodes faster than Q3_K_M (14.7 vs 13.2 tok/s, both 0-spill). It drops non-expert

infra to Q4_K, keeps experts as fast Q2_K k-quants (only the 13 imatrix-coldest layers go to

IQ2_XXS), 2.59 bpw. The trade is real quality loss — see the perplexity table — so it's for rigs

that are HBM-limited to eight cards, or that want maximum decode throughput and can tolerate Q2-level

quality. Run it with -sm layer (not -sm tensor).

---

GLM-5.2-Q4_K_XL — composition

The simplest recipe in the set and the highest quality: every routed expert at Q4_K, everything

else at Q8_0. It's the direct one-tier-up sibling of Q3_K_XL (identical recipe, q3_K→q4_K on the

experts, same wikitext imatrix), built for big-RAM CPU-expert inference — a box with ~512 GB where the

routed experts live in system RAM and a single modest GPU (or none) handles the rest.

| Tensor group | Quant |

|---|---|

| Routed experts (ffn_{gate,up,down}_exps), all MoE layers incl. blk.78 (MTP head) | Q4_K |

| MLA attention (q_a, q_b, kv_a_mqa, k_b, v_b, output) + indexer | Q8_0 |

| Shared expert (ffn_{gate,up,down}_shexp) — fires every token | Q8_0 |

| token_embd, output, dense FFN (blk.0–2) | Q8_0 |

| norms, router (ffn_gate_inp, exp_probs_b) | F32 |

Total: 403.8 GiB, 4.60 bpw, 10 shards. imatrix-quantized (wikitext-2, 200 × 512-tok) applied to the

experts — the same importance matrix as Q3_K_XL. Keeping every non-expert tensor at Q8_0 (rather than

the Q6_K the GPU-fit builds use) spends bytes on quality, since a 512 GB host has the room. Fits a 512 GB

box with headroom for a long KV cache; too large for 10×32 GB GPU 0-spill — use Q3_K_M for that.

---

GLM-5.2-Q3_K_M — composition

Built from the unsloth BF16 of GLM-5.2 (unsloth/GLM-5.2-GGUF)

using unsloth's importance matrix (see imatrix note below). Base ftype Q3_K_M, with explicit

per-tensor overrides:

| Tensor group | Quant | Notes |

|---|---|---|

| Routed experts, blk.35–77 (43 hottest MoE layers) | Q3_K | Q3_K_M base; some late-layer ffn_down_exps auto-promoted to Q4_K |

| Routed experts, blk.3–34 (32 coldest MoE layers) | Q2_K | the cold cut — see below |

| Routed experts, blk.78 (MTP / nextn head) | Q2_K | head is unused in normal decode |

| Attention (q_a, q_b, kv_a_mqa, v_b, output) | Q6_K | down from Q8_0 |

| Shared expert (ffn_{gate,up,down}_shexp) | Q6_K | fires every token |

| token_embd, output | Q6_K | |

| attn_k_b | Q4_0 | ncols=192 not divisible by 256 → automatic fallback (harmless) |

Total: 295.71 GiB, 3.37 bpw, 8 shards.

Which layers were dropped to Q2_K, and why

Only the routed experts of the 32 coldest MoE layers (blk.3–blk.34) were dropped to Q2_K

(plus the unused blk.78 MTP head). Blocks 0–2 are dense (no routed experts); blocks 35–77 — the

hottest 43 MoE layers — stay at full Q3_K.

"Cold" is measured directly from unsloth's imatrix in_sum2 (the sum of squared activations

per input channel — an importance signal). The mean ffn_down_exps importance is **monotonic in

depth and spans ~6 orders of magnitude**:

blk.3   ≈ 0.003      ← coldest
blk.20  ≈ 2.3
blk.34  ≈ 75
blk.39  ≈ 140
blk.77  ≈ 16022      ← hottest

The coldest 32 layers all sit below importance ≈ 75; the hot tail kept at Q3_K runs 88 → 16,022.

Because the early layers carry 100–1000× less activation energy than the late ones, dropping

their experts to 2-bit costs almost nothing in quality (measured below) while reclaiming the ~17 GiB

needed to fit ten cards. The MoE granularity constraint (one quant type per expert tensor — all 256

experts of a layer share it) means the cut is per-(layer, projection); the imatrix already protects

the most-used experts within each tensor.

---

GLM-5.2-Q2_K_XL — composition

Same BF16 source and unsloth imatrix. Built to cross under the 256 GB (8 × 32 GB) HBM budget at

0-spill, so every byte was cut to the floor that still runs as fast K-quants on gfx908:

| Tensor group | Quant | Notes |

|---|---|---|

| Routed experts, all MoE layers except the cold cut | Q2_K | 2-bit k-quant — still fast to dequant, unlike i-quants |

| Routed experts, 13 imatrix-coldest layers (blk.3,4,5,6,7,9–16) | IQ2_XXS | the only i-quant tier; these layers carry ~100–1000× less activation energy |

| Routed experts, blk.78 (MTP / nextn head) | IQ2_XXS | unused in normal decode |

| Attention (q_a, q_b, kv_a_mqa, v_b, output) + indexer | Q4_K | down from Q6_K |

| Shared expert (ffn_{gate,up,down}_shexp) | Q4_K | fires every token |

| token_embd, output | Q4_K | |

| attn_k_b | Q4_0 | ncols=192 not divisible by 256 → automatic fallback (harmless) |

Total: 226.9 GiB, 2.59 bpw, 6 shards. The IQ2_XXS layers are the one place this build breaks the

"K-quants only" rule — but only on the 13 coldest layers, where the dequant cost is paid rarely and

the 2.06-bpw saving is what lets the model clear the 8-card budget.

---

Quality (perplexity)

wikitext-2, 100 chunks @ 512 ctx, -fa on -ctk q8_0 -ctv q8_0, identical settings for all three:

| Quant | Size | bpw | PPL ↓ | vs this quant |

|---|---|---|---|---|

| GLM-5.2-Q4_K_XL | 403.8 GiB | 4.60 / Q8 | 2.6733 | best quality (−5.7% vs Q3_K_M, +108 GiB) |

| GLM-5.2-Q3_K_XL | 312.9 GiB | ~3.4 / Q8 | 2.8176 | −0.6% (reference, +17 GiB) |

| GLM-5.2-Q3_K_M | 295.7 GiB | 3.37 | 2.8348 | — |

| unsloth UD-IQ3_XXS | 262.3 GiB | ~3.06 | 3.2635 | +15% (−33 GiB, i-quant) |

| GLM-5.2-Q2_K_XL | 226.9 GiB | 2.59 | 3.6129 | +27% (−69 GiB) |

• Q4_K_XL is the quality ceiling of the family: at 2.6733 it sits 0.14 below Q3_K_XL

(−5% perplexity) — the payoff for taking the routed experts Q3_K → Q4_K. It costs +91 GiB over

Q3_K_XL and won't fit ten 32 GB cards 0-spill; it's for big-RAM CPU-expert hosts (≈512 GB) that

want the best quality this set offers.

• vs the full-quality Q3_K_XL: +0.6% perplexity for −17 GiB — near-free, and the 17 GiB

is what enables 0-spill on 10 cards.

• vs unsloth's similarly-positioned UD-IQ3_XXS: this quant is 13% lower perplexity while

also being a K-quant (far faster to dequant on gfx908). It spends 33 GiB more to get there — a

deliberate trade for this hardware class.

• Q2_K_XL is the quality floor: +27% perplexity vs Q3_K_M (and ~+11% vs UD-IQ3_XXS, which it

undercuts by 35 GiB and in dequant speed). It exists to buy the **8-card 0-spill fit and the fastest

decode** — pick it only when those matter more than quality. For best quality-per-byte, use Q3_K_M.

(Full 200-chunk perplexity for Q3_K_M = 3.1408 ± 0.029; the 100-chunk figures above are the

matched cross-quant comparison. Q4_K_XL = 2.6733 ± 0.033 over 100 chunks.)

---

Performance — 10 × MI100 (gfx908), 0-spill

-sm layer -fit on -fa on -ctk q8_0 -ctv q8_0 -c 8192, all 295.7 GiB resident on GPU (298.9 GiB total

VRAM incl. KV + compute; zero CPU expert spill):

| | Q3_K_M (this) |

|---|---|

| Decode | 13.2 tok/s |

| Prefill (3.5k-token prompt) | 101 tok/s default → 147 tok/s at -ub 2048 |

| CPU expert spill | 0 |

For contrast, Q3_K_XL at 312.9 GiB does not fit the same ten cards — it spills ~12 GiB of

experts back to CPU, and that spilled expert dequant is the decode bottleneck. The whole point of

Q3_K_M is to cross under the HBM budget so nothing spills.

> -ub 2048 is a free +45% on prefill here purely because the 0-spill fit leaves ~20 GiB of VRAM

> headroom to spend on a bigger compute batch.

GLM-5.2-Q2_K_XL — 8 × MI100 (gfx908), 0-spill

-sm layer -fit on -fa on -ctk q8_0 -ctv q8_0, all 226.9 GiB resident across eight 32 GB cards

(zero CPU spill):

| | Q2_K_XL |

|---|---|

| Decode | 14.7 tok/s — faster than Q3_K_M's 13.2 (fewer bytes/token) |

| Decode + ngram-simple | up to 21.9 tok/s on repetitive / templated output (accept 1.0) |

| CPU expert spill | 0 |

Q2_K_XL is the throughput pick: it fits fewer cards (8 vs 10) and decodes faster, paying for it

in quality (perplexity table above).

Speculative decoding — use ngram-simple, not ngram-cache

Model-free n-gram self-speculation, measured on the 0-spill 10×MI100 setup (decode tok/s):

| --spec-type | prose | structured code | verbatim repeat |

|---|---|---|---|

| (none) | 13.1 | 13.0 | 13.1 |

| ngram-simple ✅ | 13.1 | 15.9 (+22%) | 16.6 (+27%) |

| ngram-cache | 8.8 (−33%) | 10.3 (−21%) | 17.9 (+37%) |

Use ngram-simple — it is safe to leave on. It only drafts on a high-confidence n-gram match

(accept rate ≈ 1.0 when it fires), so it abstains on unpredictable prose with zero overhead and

speeds up structured / repetitive output. Draft depth (--spec-draft-n-max) barely matters here —

accept is already ~1.0, so 1–4 are within noise.

Tuning repetition coverage — --spec-ngram-simple-size-n (default 12). The default only fires on

a confident 12-token match, so it catches tight repetition (it abstains on looser patterns). Lower

it to fire more aggressively, at the risk of mis-drafting semi-structured text. Measured on Q2_K_XL

(decode tok/s):

| size_n | prose | log-line / templated repeat | free-form code |

|---|---|---|---|

| 12 (default, safe) | 14.7 (abstain) | 14.5 (abstain) | 14.6 (abstain) |

| 8 (moderate) | 14.6 (abstain) | 19.7 (+36%) | 12.6 (−14%) |

| 6 (aggressive) | 14.7 (abstain) | 20.2 (+37%) | 7.1 (−51%) |

Prose stays neutral at every setting. Keep 12 for mixed / code-heavy serving; drop to 8 for

agentic / log / templated workloads. size_n=6 over-drafts repeating code structure (def get_X

with varying field names → 3% accept) and is only worth it on known highly-repetitive output.

Avoid ngram-cache for general serving: it drafts aggressively on weak matches (accept ~0.13 prose /

~0.37 code), and on this MoE every rejected draft still routes its 8 experts through the verify batch,

so the wasted expert-GEMM costs more than it saves except on near-verbatim echo. Why spec is

expensive on MoE at all: the verify batch routes every drafted position to its own experts, so

wasted drafts cost real GPU compute — speculation here is gated by accept rate, not by spill.

---

Usage

# Q4_K_XL — big-RAM CPU-expert box (≈512 GB). Routed experts stay in system RAM;
# attention + KV + shared expert go on one modest GPU. Loads the 10-shard set from the first shard.
llama-server -m GLM-5.2-Q4_K_XL-00001-of-00010.gguf \
  -ngl 999 -ot 'ffn_.*_exps=CPU' -fa on -ctk q8_0 -ctv q8_0 -c 16384
# no GPU? use -ngl 0 for all-CPU. On dual-socket hosts prefix with `numactl --interleave=all`.

# Q3_K_M — llama.cpp (mainline). Loads the 8-shard set from the first shard.
llama-server -m GLM-5.2-Q3_K_M-00001-of-00008.gguf \
  -sm layer -fit on -fa on -ctk q8_0 -ctv q8_0 -c 8192 -ub 2048 \
  --spec-type ngram-simple        # safe always-on; +20-27% on structured/repetitive, 0 on prose

# Q2_K_XL — fits 0-spill on eight 32 GB cards. Loads the 6-shard set from the first shard.
HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
llama-server -m GLM-5.2-Q2_K_XL-00001-of-00006.gguf \
  -sm layer -fit on -fa on -ctk q8_0 -ctv q8_0 -c 8192 \
  --spec-type ngram-simple        # for log/templated workloads add: --spec-ngram-simple-size-n 8

Q3_K_M needs ~296 GiB of combined HBM for 0-spill (e.g. 10 × 32 GB); Q2_K_XL needs ~228 GiB (e.g.

8 × 32 GB). With less, -fit auto-spills the coldest experts to CPU and it still runs — just slower

on the spilled layers. Use -sm layer (not -sm tensor) for both.

Credits / provenance

• Base model: GLM-5.2 by zai-org.
• Source weights (BF16) and the importance matrix used for this quant are from

unsloth/GLM-5.2-GGUF — the imatrix is not

redistributed here; grab it from unsloth's repo if you want to reproduce. Thanks to the unsloth

team for the calibration data.

• Quantized with llama.cpp llama-quantize

(--imatrix + --tensor-type-file).

• Provenance differs by build: Q3_K_M / Q2_K_XL use the unsloth BF16 + unsloth imatrix

above. The _XL builds (Q3_K_XL, Q4_K_XL) instead use a wikitext-2 imatrix and a plain

convert_hf_to_gguf.py → Q8_0 intermediate, with base type Q8_0 and experts overridden via

--custom-q 'ffn_.*_exps=q{3,4}_K' — i.e. everything Q8_0 except the routed experts.