mudler/qwen3.5-35b-a3b-apex-tq-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "base_model": "Qwen/Qwen3.5-35B-A3B",
    "tags": [
      "gguf",
      "quantized",
      "apex",
      "moe",
      "mixture-of-experts",
      "qwen3.5",
      "turboquant",
      "experimental"
    ],
    "frontmatter": {
      "license": "apache-2.0",
      "base_model": "Qwen/Qwen3.5-35B-A3B",
      "tags": [
        "gguf",
        "quantized",
        "apex",
        "moe",
        "mixture-of-experts",
        "qwen3.5",
        "turboquant",
        "experimental"
      ]
    },
    "hero_image_url": "",
    "summary": "> **WARNING: These GGUFs require TurboQuant Plus — a fork of llama.cpp with TQ4_1S support. They will NOT work with standard llama.cpp.** **APEX-TQ (Adaptive Precision for EXpert Models + TurboQuant)** experimental quantizations of Qwen3.5-35B-A3B. **Brought to you by the LocalAI team** | APEX Project | Technical Report",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nbase_model: Qwen/Qwen3.5-35B-A3B\ntags:\n  - gguf\n  - quantized\n  - apex\n  - moe\n  - mixture-of-experts\n  - qwen3.5\n  - turboquant\n  - experimental\n---\n\n# Qwen3.5-35B-A3B APEX-TQ GGUF (Experimental)\n\n> **WARNING: These GGUFs require [TurboQuant Plus](https://github.com/TheTom/turboquant_plus) — a fork of llama.cpp with TQ4_1S support. They will NOT work with standard llama.cpp.**\n\n**APEX-TQ (Adaptive Precision for EXpert Models + TurboQuant)** experimental quantizations of [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B).\n\n**Brought to you by the [LocalAI](https://github.com/mudler/LocalAI) team** | [APEX Project](https://github.com/mudler/apex-quant) | [Technical Report](https://github.com/mudler/apex-quant/blob/main/paper/APEX_Technical_Report.pdf)\n\n## What is APEX-TQ?\n\nAPEX-TQ combines the APEX layer-wise precision gradient with TurboQuant's TQ4_1S format, which is optimized for fast prompt processing. The result trades ~1% perplexity for **3x faster prompt processing speed** compared to standard K-quant formats.\n\n**This is experimental.** TQ4_1S is not yet merged into mainline llama.cpp. You must build from the [TurboQuant Plus fork](https://github.com/nicebyte/llama.cpp/tree/turboquant-plus).\n\n## Benchmark Results\n\n| Configuration | Size (GB) | Perplexity | pp512 (t/s) | tg128 (t/s) |\n|---|---|---|---|---|\n| APEX-TQ Quality | ~21 GB | 6.614 | **5,572** | 68.4 |\n| APEX-TQ Balanced | ~23 GB | 6.622 | **5,218** | 65.2 |\n| APEX-TQ Compact | ~16 GB | 6.833 | **4,890** | 71.1 |\n| APEX Quality (standard) | 21.3 GB | 6.527 | 1,861 | 68.4 |\n\npp512 speeds are **3x faster** than standard APEX due to TurboQuant's optimized GEMM kernels.\n\n## Available Files\n\n| File | Profile | Size | Best For |\n|------|---------|------|----------|\n| Qwen3.5-35B-A3B-APEX-TQ-Quality.gguf | TQ Quality | ~21 GB | Best quality with TQ speed boost |\n| Qwen3.5-35B-A3B-APEX-TQ-Balanced.gguf | TQ Balanced | ~23 GB | General purpose + fast prompts |\n| Qwen3.5-35B-A3B-APEX-TQ-Compact.gguf | TQ Compact | ~16 GB | Smallest TQ variant |\n\n## How to Use\n\n**You must use the TurboQuant Plus fork of llama.cpp:**\n\n```bash\ngit clone https://github.com/nicebyte/llama.cpp -b turboquant-plus\ncd llama.cpp\ncmake -B build -DGGML_CUDA=ON\ncmake --build build --config Release -j $(nproc)\n\n# Run\n./build/bin/llama-cli -m Qwen3.5-35B-A3B-APEX-TQ-Quality.gguf -ngl 99 -p \"Hello\"\n```\n\nFor standard llama.cpp compatible APEX quants, use [mudler/Qwen3.5-35B-A3B-APEX-GGUF](https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF) instead.\n\n## Architecture\n\n- **Model**: Qwen3.5-35B-A3B\n- **Layers**: 40\n- **Experts**: 256 routed + 1 shared (8 active per token)\n- **Total Parameters**: ~35B\n- **Active Parameters**: ~3B per token\n- **Quantization**: APEX layer gradient + TQ4_1S format for expert weights\n\n## Credits\n\nAPEX is brought to you by the [LocalAI](https://github.com/mudler/LocalAI) team. TurboQuant by [TheTom](https://github.com/TheTom/turboquant_plus). Built on [llama.cpp](https://github.com/ggerganov/llama.cpp).\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "quantized",
    "apex",
    "moe",
    "mixture-of-experts",
    "qwen3.5",
    "turboquant",
    "experimental",
    "base_model:Qwen/Qwen3.5-35B-A3B",
    "base_model:quantized:Qwen/Qwen3.5-35B-A3B",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "imatrix",
    "conversational"
  ],
  "likes": 15,
  "downloads": 4097,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-05T07:28:28.000Z",
  "created_at": "2026-04-03T08:56:51.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "69cf80d3a91383ae4eaa5e48",
  "id": "mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF",
  "modelId": "mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF",
  "sha": "50b47997d43433a4800e7cdacd42bfb2ebf9aad9",
  "createdAt": "2026-04-03T08:56:51.000Z",
  "lastModified": "2026-04-05T07:28:28.000Z",
  "author": "mudler",
  "downloads": 4097,
  "likes": 15,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 5
}