---

license: apache-2.0

library_name: llama.cpp

tags:

• quantized
• moe
• glm
• mixed-precision
• sharded
• gguf-split
• epyc
• rtx-5090

---

GLM-5.2-BF16 → mxfp4_moe (Mixed-Precision Quantization + Sharded)

This repository contains a quantized and sharded version of the GLM-5.2 base model (originally in BF16), converted to the GGUF format using llama.cpp.

To make the model both efficient and practical for deployment, I applied two distinct processing steps:

1. Mixed-Precision Quantization – to aggressively reduce file size while preserving critical quality.
2. Sharding (Splitting) – to break the large model into smaller, filesystem-friendly chunks (e.g., to comply with Hugging Face's 6 GB file limit or to enable resumable downloads).

---

1. Quantization Strategy

To achieve the best balance between model size and generation quality, I used a specialized mixed-precision approach with the llama-quantize tool.

This model uses:

• mxfp4_moe for the majority of the tensors (especially the MoE expert layers). This is a specialized 4-bit floating-point format designed to efficiently compress Mixture-of-Experts (MoE) models while preserving reasoning capabilities.
• Q8_0 specifically for the embedding table (token_embd.weight) and the output layer (output.weight). These layers are notoriously sensitive to quantization—keeping them at higher precision (8-bit) significantly reduces degradation in output quality, especially for long generations and complex reasoning tasks.

Quantization Command

The exact command used to generate the base quantized file was:

./llama-quantize \
  --allow-requantize \
  --tensor-type token_embd.weight=Q8_0 \
  --tensor-type output.weight=Q8_0 \
  GLM-5.2-BF16.gguf \
  glm-5.2-mxfp4-moe-q8.gguf \
  mxfp4_moe

---

Performance Benchmarks

Inference speed was evaluated on the following enterprise-grade hardware configuration:

• Motherboard: Gigabyte MZ73-LM0
• CPU: AMD EPYC 9654 (96 cores / 192 threads, Genoa architecture)
• Memory: 768 GB DDR5 (12-channel configuration, fully populated)
• GPU: NVIDIA RTX 5090 (32 GB VRAM)

Under this setup, the quantized model achieves a generation speed of:

> ≈ 6.8 tokens per second (t/s)

This benchmark was recorded using llama-server with standard inference settings (batch size = 512, context length = 2048, with layers offloaded to the GPU via -ngl). The combination of the EPYC 9654's massive memory bandwidth (over 400 GB/s thanks to 12-channel DDR5) and the RTX 5090's computational muscle allows the MoE architecture to run smoothly, fully leveraging the mixed-precision quantization.

At 6.8 t/s, the model provides a fluid, interactive experience suitable for real-time chat applications, complex multi-turn dialogues, and light batch processing—delivering near-instantaneous responses without frustrating delays.

./llama-server -m /path/GLM-5.2-MXFP4-MOE-Q8_0.GGUF -ngl 99 -ot "exps=CPU" -t 80 -tb 96 --host xxx.xxx.xxx.xxx --port 8080 --ctx-size 98304 --batch-size 2048 --ubatch-size 2048 --no-mmap --embeddings --pooling mean -fa on