---

license: apache-2.0

library_name: llama.cpp

tags:

• gguf
• moe
• xing4_0
• apex-quant
• mixture-of-experts
• multi-latent-attention

pipeline_tag: text-generation

base_model:

• XingChen-AGI/Xing4.0-29B-A4B

---

Xing4.0-29B-A4B — Apex Quant GGUF Models

This repository contains 3 quantized GGUF profiles of the XingChen-AGI/Xing4.0-29B-A4B model, produced using Apex-Quant mixed-precision quantization technology.

> Note: The original source model is available at XingChen-AGI/Xing4.0-29B-A4B under Apache 2.0 license. This repository provides quantized variants only.

📦 Quick Overview

| Profile | Size | BPW | Best For |

|---------|------|-----|----------|

| i-quality | 19.21 GB | ~5.2 | Highest fidelity, production use, research |

| i-balanced | 22 GB | ~5.5 | Balanced quality/performance trade-off |

| i-compact | 14 GB | 3.85 | Resource-constrained deployment, fast inference |

> BPW = Bits Per Weight. Higher value generally correlates with better generation quality.

📁 Repository Contents

├── Xing4.0-29B-A4B-i-quality.gguf   # 19.21 GB — highest quality
├── Xing4.0-29B-A4B-i-balanced.gguf  # 22 GB — balanced
├── Xing4.0-29B-A4B-i-compact.gguf   # 14 GB — compact
└── patches/
    └── 0001-xing4_0-llama-cpp-build.patch  # Reproducible build instructions

🏗️ Source Model Information

| Property | Value |

|----------|-------|

| Architecture | Xing4_0ForCausalLM |

| Total Parameters | 29B (4B active per token) |

| Layers | 40 (2 dense lead-in + 38 MoE blocks) |

| Hidden Size | 3,584 |

| Vocabulary | 131,072 tokens (SentencePiece) |

| Context Length | 262,144 tokens (YaRN extended from 4,096) |

| License | Apache-2.0 |

| Source | China Telecom / XingChen-AGI |

Key Architecture Features

• Multi-Latent Attention (MLA): DeepSeek-V3-style MLA compression with q_lora_rank=768, kv_lora_rank=512, enabling efficient attention without full projection matrices.
• Mixture-of-Experts (MoE): 64 routed experts per layer, top-4 active per token (noaux_tc gating method), plus 1 shared expert per layer.
• Masked Hyper-Connections (mHC): Sinkhorn-gated routing (mh_mult=4, sinkhorn_iters=20) for enhanced expert collaboration.
• Multi-Token Prediction (MTP): Draft head at layer 40 supporting speculative decoding.
• RoPE: YaRN positional embeddings with scaling factor 64, theta 10,000.

🔬 Quantization Methodology

All three profiles were generated using the Apex-Quant framework with importance-weighted mixed-precision quantization:

Apex-Quant Configuration

# Generate per-profile config (EDGE/NEAR/MID layer bands)
bash scripts/generate_config.sh \
  --profile i-quality \
  --layers 40 \
  --dense-layers 2 > config.txt

# Append MTP block patterns (blk.40) for draft-head quantization
echo "blk.40.ffn_gate_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_up_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_down_exps=iq4_xs" >> config.txt
# ... (see config files for full content)

# Quantize with imatrix
llama-quantize \
  --tensor-type-file config.txt \
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf \
  Xing4.0-29B-A4B-BF16.gguf \
  Xing4.0-29B-A4B-i-quality.gguf \
  Q6_K

Layer Band Allocation

| Region | Layers | i-quality Experts | i-balanced Experts | i-compact Experts |

|--------|--------|-------------------|--------------------|-------------------|

| EDGE | L0–4, L35–39 | Q6_K | Q6_K | Q4_K |

| NEAR | L5–9, L30–34 | Q5_K | Q5_K | Q3_K |

| MID | L10–29 | IQ4_XS | Q5_K | Q3_K |

| Shared FFN | All layers | Q8_0 | Q8_0 | Q6_K |

| Attention | L0–2, L37–39 | Q6_K | Q6_K | Q4_K |

| Attention | L3–36 | Q6_K | Q6_K | Q4_K |

Important Implementation Notes

1. MTP Block (blk.40): The multi-token prediction draft head has no imatrix entries because it is not executed during trunk-only inference. Types requiring imatrix (IQ1/IQ2 series) cannot be used for blk.40 tensors — this profile uses IQ4_XS/Q5_K/Q3_K which do not require imatrix.

1. mHC Gate Tensors: Automatically excluded from quantization by the llama.cpp fork (hc_attn_fn.weight, hc_ffn_fn.weight, hc_head_fn.weight preserved at F32).

1. MLA B-Matrices (attn_k_b): Tensors with shape [128, 512, 32] have an inner dimension (128) that is not divisible by Q6_K's block size (256). These automatically fall back to Q8_0 via the built-in fallback mechanism — a safe direction (higher precision).

1. Router Tensors (ffn_gate_inp): Fall through to the base type (Q6_K for i-quality, Q5_K for i-balanced, Q4_K for i-compact).

1. Config File Format: The --tensor-type-file parser does NOT support comment lines (#). Only pure pattern=value lines are valid.

💻 Inference Usage

Recommended Settings (from official repo)

[main]
ctx-size = 65536
flash-attn = true
cache-type-k = q8_0
cache-type-v = q8_0
n-gpu-layers = 999

llama.cpp CLI Examples

# i-compact — fastest, lowest VRAM requirement
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --top-k 50 --top-p 0.95 ^
  -p "Türkiye'nin en büyük şehri neresidir?" -n 64

# i-quality — highest accuracy
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-quality.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --seed 42 ^
  -p "Explain quantum computing in simple terms." -n 64

# Server mode
.\llama-server.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --host 0.0.0.0 --port 8080

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Xing4.0-29B-A4B-i-compact.gguf",
    n_ctx=65536,
    n_gpu_layers=999,
    flash_attn=True,
    seed=42,
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello, tell me about Istanbul."}],
    max_tokens=256,
)
print(response["choices"][0]["message"]["content"])

VRAM Sizing Guide

| VRAM | Profile | Offload Strategy |

|------|---------|-----------------|

| 8 GB | i-compact | Partial (-ngl 6–10), expect slow CPU decode |

| 16 GB | i-compact | Full or near-full (-ngl 20–35); i-quality partial |

| 24 GB | i-compact (full), i-quality (partial) | -ngl 35–40 for i-quality |

| 40+ GB | All profiles | Full offload (-ngl 999) |

> KV Cache: At 65,536 context with q8_0/kv ≈ 2.5 GB. To save VRAM, disable GPU KV offload with --no-kv-offload (moves KV cache to system RAM).

🛠️ Reproducible Build

To reproduce these quantizations from scratch:

Step 1: Clone and Build the Xing4.0 Fork

git clone -b xing4_0-port https://github.com/shuxiaoqiong/llama.cpp.git llama-cpp-xing4
cd llama-cpp-xing4

cmake -G "Visual Studio 17 2022" -A x64 ^
      -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON ^
      -B build
cd build
cmake --build . --config Release --parallel

Required binaries: llama-quantize.exe, llama-imatrix.exe, llama-cli.exe

Step 2: Convert Source Model to GGUF

python convert_hf_to_gguf.py ^
  <path-to-safetensors-dir> ^
  --outfile Xing4.0-29B-A4B-BF16.gguf ^
  --outtype bf16

Expected output: ~59 GB GGUF with 977 tensors.

Step 3: Generate Apex Configs

# Quality profile
bash scripts/generate_config.sh --profile i-quality --layers 40 --dense-layers 2 > configs/xing4_quality.txt

# Balanced profile
bash scripts/generate_config.sh --profile i-balanced --layers 40 --dense-layers 2 > configs/xing4_balanced.txt

# Compact profile
bash scripts/generate_config.sh --profile i-compact --layers 40 --dense-layers 2 > configs/xing4_compact.txt

Append MTP block (blk.40) patterns to each file:

blk.40.ffn_gate_exps=<EXP_TYPE>
blk.40.ffn_up_exps=<EXP_TYPE>
blk.40.ffn_down_exps=<EXP_TYPE>
blk.40.ffn_gate_shexp=<SHARED_TYPE>
blk.40.ffn_up_shexp=<SHARED_TYPE>
blk.40.ffn_down_shexp=<SHARED_TYPE>
blk.40.attn_q=<ATTN_TYPE>
blk.40.attn_k=<ATTN_TYPE>
blk.40.attn_v=<ATTN_TYPE>
blk.40.attn_output=<ATTN_TYPE>

Where <EXP_TYPE> = iq4_xs (quality) / Q5_K (balanced) / Q3_K (compact), and <ATTN_TYPE> = Q6_K (quality/balanced) / Q4_K (compact).

Step 4: Run Quantization

See the batch script at .tmp/apex_quant_xing4.ps1 for the automated 3-profile run, or invoke individually:

llama-quantize.exe ^
  --tensor-type-file configs\xing4_quality.txt ^
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf ^
  Xing4.0-29B-A4B-BF16.gguf ^
  Xing4.0-29B-A4B-i-quality.gguf ^
  Q6_K

📊 Technical Summary

Tensor Distribution Statistics

| Component | Parameter Count | Approx. Size (BF16) |

|-----------|----------------|---------------------|

| Token Embedding | 470M | 940 MB |

| Output Projection | 470M | 940 MB |

| Attention Projections (all layers) | ~1.2B | ~2.4 GB |

| Expert FFNs (routed, 64×38 layers) | ~23B | ~46 GB |

| Shared Expert FFNs (1×40 layers) | ~360M | ~720 MB |

| mHC Gates & Norms | ~50M | ~100 MB |

| MTP Draft Head (blk.40) | ~3B | ~6 GB |

| Total | ~29B | ~59 GB |

Quantization Impact

| Metric | BF16 Base | i-quality | i-balanced | i-compact |

|--------|-----------|-----------|------------|-----------|

| Total Size | 59 GB | 19.21 GB | 22 GB | 14 GB |

| Compression Ratio | 1× | 3.07× | 2.68× | 4.21× |

| Expert FFN Precision | F16 | Q6/Q5/IQ4 | Q6/Q5 | Q4/Q3 |

| Shared FFN | F16 | Q8_0 | Q8_0 | Q6_K |

| Attention | F16 | Q6_K | Q6_K | Q4_K |

🙏 Acknowledgments

• localai-org/apex-quant — Apex-Quant MoE-aware mixed-precision quantization framework
• shuxiaoqiong/llama.cpp — Xing4.0 port fork (branch xing4_0-port) with native architecture support
• ggerganov/llama.cpp — GGUF format specification and quantization engine
• XingChen-AGI — Original Xing4.0-29B-A4B model creators (China Telecom / XingChen-AGI)
• jmarceno — Community imatrix and quantization recipe reference (imatrix-Xing4.0-29B-A4B-820chunks.gguf)

📝 Known Issues & Limitations

1. Smoke Test Output: Initial smoke testing with llama-cli returned whitespace-only output on CPU (-ngl 0) and partial GPU offload (-ngl 20). This may indicate MoE expert routing issues in certain runtime configurations. Further testing with different sampler parameters (--top-k, --min-p) and full GPU offload is recommended before production use.

1. MTP Draft Head: The draft head at blk.40 is loaded but not executed during standard inference. Speculative decoding support depends on the inference engine's MTP implementation.

1. Imatrix Dependency: While IQ4_XS, Q5_K, Q4_K, and Q3_K types work without imatrix, any future profiles using IQ1/IQ2 series will fail on blk.40 tensors due to missing imatrix entries for the MTP block.

📄 License

The original model XingChen-AGI/Xing4.0-29B-A4B is distributed under the Apache-2.0 license. These quantized derivatives are shared under the same license. See the original model card for full license terms and usage restrictions.

🔗 Related Links

• Original Model: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B
• Official GGUF Repo: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B-GGUF
• Deploy Tutorial: https://github.com/shuxiaoqiong/xing4_0-llama-pc-deploy
• Apex-Quant Framework: https://github.com/localai-org/apex-quant
• llama.cpp Engine: https://github.com/ggerganov/llama.cpp
• GGUF Specification: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md