icecubetr/GLM-5.2-504B-Code-GGUF overview

---

license: mit

library_name: gguf

pipeline_tag: text-generation

tags:

• moe
• reap
• gguf
• llama.cpp
• glm
• glm-5.2
• quantized
• gguf

base_model: zai-org/GLM-5.2

base_model_relation: quantized

---

Model Description

This is a quantized version of the GLM-5.2-REAP-504B-GGUF model. The model has been converted from BF16 to GGUF format with importance matrix optimization.

Provided Quants

| file | bits | size | notes |

|---|---|---|---|

| GLM-5.2-504B-Code-IQ2_XXS.gguf | 2.29 | ~140 GB | smallest |

Other Files

| file | notes |

|---|---|

| GLM-5.2-REAP-504B-imatrix.gguf | imatrix file created with code.txt dataset, saved after 80/146 chunks of calibration data |

| llama-quant.cpp | modified llama.cpp file for leaving indexer tensors unquantized |

Run it

llama-cli -m GLM-5.2-504B-Code-IQ2_XXS.gguf -p "Hello"
llama-server -m GLM-5.2-504B-Code-IQ2_XXS.gguf --host 0.0.0.0 --port 8080

Recommended serving — recover most of the loop gap for free

• Anti-loop (recommended):

- min_p=0.05, repetition_penalty=1.05

- min_p=0.05, repetition_penalty=1.10

Start at 1.05; go to 1.10 if you see loops — a higher repetition penalty trades a little risk of

over-penalizing legitimate repetition (e.g. in code) for near-zero looping.

• Conciseness: a brevity system prompt — *"Be concise. Think only as much as the task needs, then

answer and stop." — halves median length (1267 → 507 tokens). Note it does not* reduce looping

(that's the sampler's job); combine the two for short, low-loop output.

Quantization Details

This model has been quantized using llama.cpp's llama-quantize tool with importance matrix (--imatrix) for optimal quantization. The quantization target was IQ2_XXS (2-bit quantization).

Quantization Summary

| Metric | Value |

|--------|-------|

| Original Model Size (BF16) | 956,419.32 MiB (16.00 BPW) |

| Quantized Model Size | 136,813.06 MiB (2.29 BPW) |

| Compression Ratio | ~7:1 |

| Total Tensors | 1809 |

| Tensors with Fallback | 79 |

Tensor Type to Quantization Method Mapping

| Tensor Pattern | Quantization Method | Original Format | Notes |

|----------------|---------------------|-----------------|-------|

| *attn_k_b.weight | iq4_nl | bf16 | Fallback from IQ2_XXS (ncols=192 not divisible by 256) |

| *attn_kv_a_mqa.weight | iq2_xxs | bf16 | |

| *attn_output.weight | iq2_xxs | bf16 | |

| *attn_q_a.weight | iq2_xxs | bf16 | |

| *attn_q_b.weight | iq2_xxs | bf16 | |

| *attn_v_b.weight | iq2_xxs | bf16 | |

| *ffn_down_exps.weight | iq2_xxs | bf16 | |

| *ffn_down_shexp.weight | iq2_xxs | bf16 | |

| *ffn_gate_exps.weight | iq2_xxs | bf16 | |

| *ffn_gate_shexp.weight | iq2_xxs | bf16 | |

| *ffn_up_exps.weight | iq2_xxs | bf16 | |

| *ffn_up_shexp.weight | iq2_xxs | bf16 | |

| blk.0.ffn_down.weight | q2_K | bf16 | Dense layers (blocks 0-2) |

| blk.1.ffn_down.weight | q2_K | bf16 | Dense layers (blocks 0-2) |

| blk.2.ffn_down.weight | q2_K | bf16 | Dense layers (blocks 0-2) |

| output.weight | q5_K | bf16 | Output projection |

| token_embd.weight | q2_K | bf16 | Token embeddings |

Tensors Kept in Original Format (Not Quantized)

f32 (Float 32-bit) - Kept for Numerical Stability

• All _norm.weight and _norm.bias tensors
• All *_gate_inp.weight tensors (e.g., ffn_gate_inp.weight)
• All exp_probs_b.bias tensors
• blk.78.nextn.enorm.weight
• blk.78.nextn.hnorm.weight
• blk.78.nextn.shared_head_norm.weight

bf16 (bfloat16) - Not Quantized (Missing Importance Matrix Data)

• All indexer.attn_k.weight tensors
• All indexer.attn_q_b.weight tensors
• All indexer.proj.weight tensors
• blk.78.attn_kv_a_mqa.weight
• blk.78.attn_output.weight
• blk.78.attn_q_a.weight
• blk.78.attn_q_b.weight
• blk.78.attn_v_b.weight
• blk.78.ffn_down_exps.weight
• blk.78.ffn_down_shexp.weight
• blk.78.ffn_gate_exps.weight
• blk.78.ffn_gate_shexp.weight
• blk.78.ffn_up_exps.weight
• blk.78.ffn_up_shexp.weight
• blk.78.nextn.eh_proj.weight

Quantization Method Details

| Method | Bits | Description |

|--------|------|-------------|

| IQ2_XXS | ~2 | Extremely low-bit quantization with importance matrix optimization |

| IQ4_NL | ~4 | Non-linear 4-bit quantization with higher quality |

| Q2_K | ~2 | K-quant 2-bit quantization for dense layers |

| Q5_K | ~5 | K-quant 5-bit quantization (used for output layer) |

Important Notes

1. Fallback Quantization: 79 tensors required fallback quantization because:

- attn_k_b.weight tensors have ncols=192 (not divisible by 256, required for IQ2_XXS), falling back to IQ4_NL

- Other tensors missing importance matrix data

1. Indexer Tensors: All indexer.* tensors were kept in BF16 format as they are critical for the model's routing/indexing mechanism and no importance data was available.

1. Final Layer: blk.78 (the final transformer block) was largely left unquantized to preserve output quality, as importance data was missing for these tensors.

1. Dense Layers: The first three blocks (blk.0 to blk.2) use dense FFN layers instead of MoE, which were quantized with Q2_K.

1. Importance Matrix: Quantization used an importance matrix from 80 chunks of calibration data, optimizing which tensors receive more bits based on their importance to model performance.