groxaxo/qwen3.5-122b-a10b-abliterix-ud-q4_k_xl-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "base_model": "wangzhang/Qwen3.5-122B-A10B-abliterix",
    "tags": [
      "gguf",
      "llama-cpp",
      "quantized",
      "ud-quantization",
      "user-defined",
      "mixed-precision",
      "importance-matrix",
      "abliterated",
      "uncensored",
      "qwen3.5",
      "moe"
    ],
    "language": [
      "en"
    ],
    "pipeline_tag": "text-generation",
    "frontmatter": {
      "license": "apache-2.0",
      "base_model": "wangzhang/Qwen3.5-122B-A10B-abliterix",
      "tags": [
        "gguf",
        "llama-cpp",
        "quantized",
        "ud-quantization",
        "user-defined",
        "mixed-precision",
        "importance-matrix",
        "abliterated",
        "uncensored",
        "qwen3.5",
        "moe"
      ],
      "language": [
        "en"
      ],
      "pipeline_tag": "text-generation"
    },
    "hero_image_url": "",
    "summary": "User-Defined (UD) mixed-precision quantization of Qwen3.5-122B-A10B-abliterix for llama.cpp.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nbase_model: wangzhang/Qwen3.5-122B-A10B-abliterix\ntags:\n- gguf\n- llama-cpp\n- quantized\n- ud-quantization\n- user-defined\n- mixed-precision\n- importance-matrix\n- abliterated\n- uncensored\n- qwen3.5\n- moe\nlanguage:\n- en\npipeline_tag: text-generation\n---\n\n# Qwen3.5-122B-A10B-abliterix UD-Q4_K_XL GGUF\n\nUser-Defined (UD) mixed-precision quantization of [Qwen3.5-122B-A10B-abliterix](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterix) for llama.cpp.\n\n## What is UD (User-Defined) Quantization?\n\nUD quantization is a **per-tensor mixed-precision** approach that goes beyond blanket quantization methods (like Q4_K_M or Q8_0 applied uniformly). Instead of using the same quantization type for every tensor, UD assigns an **individually chosen type** to each tensor based on its role in the model architecture and its sensitivity to precision loss.\n\n### How it differs from standard quantization\n\n| Aspect | Standard (e.g. Q4_K_M) | UD Quantization |\n|---|---|---|\n| Tensor types | One type for all (or a fixed pattern) | Each tensor gets its own type |\n| Precision allocation | Uniform | Adaptive per tensor role |\n| Importance matrix | Optional (general guidance) | Used to optimize k-quant block selection |\n| Expert tensors | Same type as all others | Can be individually tuned |\n\n### The UD methodology for this model\n\n1. **Architecture analysis**: Qwen3.5-122B-A10B is a Mixture-of-Experts (MoE) model with 48 layers, 256 experts per layer (8 active), shared expert FFN, and SSM (state-space model) components. Each tensor was classified by its function:\n\n   - **Attention weights** (`attn_qkv`, `attn_gate`, `attn_q`, `attn_k`, `attn_v`, `attn_output`): Quantized to **Q8_0** — attention projections are highly sensitive to precision loss and benefit from near-lossless compression.\n   \n   - **Expert FFN up/gate weights** (`ffn_up_exps`, `ffn_gate_exps`): Quantized to **Q4_K** — these are the largest tensors in the model (256 experts x 3072x1024 each) and dominate the model size. Q4_K provides an excellent size/quality tradeoff for activation functions.\n   \n   - **Expert FFN down weights** (`ffn_down_exps`): Quantized to **Q5_K** — the down-projection is slightly more sensitive than up/gate, so Q5_K preserves more information here.\n   \n   - **Shared expert FFN** (`ffn_down_shexp`, `ffn_up_shexp`, `ffn_gate_shexp`, `ffn_gate_inp_shexp`): Quantized to **Q8_0** — the shared expert processes every token, making it critical for quality.\n   \n   - **Router/gating** (`ffn_gate_inp`, `ffn_gate_inp_shexp`): Kept at **F32** — the router decides which experts to activate. Even small precision errors here can cause misrouting, severely degrading output quality.\n   \n   - **SSM components** (`ssm_a`, `ssm_conv1d`, `ssm_dt.bias`, `ssm_norm`): Kept at **F32** or **Q8_0** — these are small tensors that play a structural role in the hybrid attention/SSM layers.\n   \n   - **Norms** (`attn_norm`, `post_attention_norm`, `output_norm`): Kept at **F32** — normalization layers are tiny but critical for numerical stability.\n   \n   - **Token embeddings** (`token_embd.weight`): Quantized to **Q8_0**.\n   \n   - **Output projection** (`output.weight`): Quantized to **Q8_0**.\n\n2. **Importance matrix**: A partial importance matrix (imatrix) was computed on coding-focused calibration data using `llama-imatrix`. This guides the k-quant block selection within each tensor, prioritizing precision on the most activation-sensitive dimensions. The imatrix was computed at `n_ctx=512` across multiple calibration chunks before the quantization run.\n\n3. **Quantization**: `llama-quantize` from [llama.cpp](https://github.com/ggml-org/llama.cpp) was used with `--tensor-type-file` to apply the per-tensor type map, combined with `--imatrix` for importance-weighted block selection within each quantized tensor.\n\n### Resulting size\n\n| Component | Size |\n|---|---|\n| This GGUF file | ~72 GB |\n| Original BF16 GGUF | ~244 GB |\n| Compression ratio | ~3.4x |\n\n## Usage with llama.cpp\n\n```bash\nllama-server -m Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL.gguf \\\n  -ngl 999 \\\n  -c 32768 \\\n  -fa on \\\n  --reasoning auto\n```\n\n### Multi-GPU tensor split example (3x RTX 3090 + 2x RTX 3060)\n\n```bash\nllama-server -m Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL.gguf \\\n  -ngl 999 \\\n  -ts 24,24,12,12,24 \\\n  -c 32768 \\\n  -ctk q8_0 -ctv q8_0 \\\n  -fa on \\\n  --reasoning auto\n```\n\n## Provided by\n\n- **Quantized by**: [groxaxo](https://huggingface.co/groxaxo)\n- **Base model**: [wangzhang/Qwen3.5-122B-A10B-abliterix](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterix)\n- **Quantization engine**: [llama.cpp](https://github.com/ggml-org/llama.cpp) build 8736\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "llama-cpp",
    "quantized",
    "ud-quantization",
    "user-defined",
    "mixed-precision",
    "importance-matrix",
    "abliterated",
    "uncensored",
    "qwen3.5",
    "moe",
    "text-generation",
    "en",
    "base_model:wangzhang/Qwen3.5-122B-A10B-abliterix",
    "base_model:quantized:wangzhang/Qwen3.5-122B-A10B-abliterix",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "imatrix",
    "conversational"
  ],
  "likes": 1,
  "downloads": 396,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-14T07:00:46.000Z",
  "created_at": "2026-04-14T01:58:05.000Z",
  "pipeline_tag": "text-generation",
  "library_name": ""
}

{
  "_id": "69dd9f2d96792db3a239f0c4",
  "id": "groxaxo/Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL-GGUF",
  "modelId": "groxaxo/Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL-GGUF",
  "sha": "5d5ebdacc9db8333c429c4b4564ba5c88362e743",
  "createdAt": "2026-04-14T01:58:05.000Z",
  "lastModified": "2026-04-14T07:00:46.000Z",
  "author": "groxaxo",
  "downloads": 396,
  "likes": 1,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "",
  "siblings_count": 4
}