infatoshi/qwen3.6-35b-a3b-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "base_model": "Qwen/Qwen3.6-35B-A3B",
    "base_model_relation": "quantized",
    "language": [
      "en",
      "zh",
      "multilingual"
    ],
    "pipeline_tag": "image-text-to-text",
    "library_name": "gguf",
    "tags": [
      "gguf",
      "llama.cpp",
      "qwen3_5_moe",
      "multimodal",
      "vision",
      "moe"
    ],
    "frontmatter": {
      "license": "apache-2.0",
      "base_model": "Qwen/Qwen3.6-35B-A3B",
      "base_model_relation": "quantized",
      "language": [
        "en",
        "zh",
        "multilingual"
      ],
      "pipeline_tag": "image-text-to-text",
      "library_name": "gguf",
      "tags": [
        "gguf",
        "llama.cpp",
        "qwen3_5_moe",
        "multimodal",
        "vision",
        "moe"
      ]
    },
    "hero_image_url": "",
    "summary": "GGUF conversions of Qwen/Qwen3.6-35B-A3B for llama.cpp. Sparse MoE: 35B total / 3B active (256 experts + 1 shared, top-8 routed). Hybrid Gated-DeltaNet + Gated-Attention layers (3:1), 262K native context. Includes vision projector for image/video input.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nbase_model: Qwen/Qwen3.6-35B-A3B\nbase_model_relation: quantized\nlanguage:\n- en\n- zh\n- multilingual\npipeline_tag: image-text-to-text\nlibrary_name: gguf\ntags:\n- gguf\n- llama.cpp\n- qwen3_5_moe\n- multimodal\n- vision\n- moe\n---\n\n# Qwen3.6-35B-A3B-GGUF\n\nGGUF conversions of [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) for llama.cpp.\n\nSparse MoE: 35B total / 3B active (256 experts + 1 shared, top-8 routed). Hybrid Gated-DeltaNet + Gated-Attention layers (3:1), 262K native context. Includes vision projector for image/video input.\n\n## Files\n\n| File | Size | Target HW |\n|---|---|---|\n| `Qwen3.6-35B-A3B-Q4_K_M.gguf` | 20 GB | Single 24GB GPU (3090/4090/5090/A5000) |\n| `Qwen3.6-35B-A3B-Q5_K_M.gguf` | 23 GB | 32GB+ VRAM or partial offload |\n| `Qwen3.6-35B-A3B-Q8_0.gguf` | 34 GB | 40GB+ VRAM (A6000/A100) or CPU |\n| `Qwen3.6-35B-A3B-BF16.gguf` | 65 GB | CPU or multi-GPU |\n| `Qwen3.6-35B-A3B-mmproj-BF16.gguf` | 862 MB | Required for vision input |\n\n## Benchmarks (RTX 3090, bs=1, llama.cpp build b1-94ca829)\n\n| Quant | Offload | Prefill | Decode | wikitext-2-raw PPL |\n|---|---|---|---|---|\n| Q4_K_M | 41/41 GPU | 329.7 t/s | 153.9 t/s | 6.676 ± 0.043 |\n| Q5_K_M | 36/41 GPU | 159.9 t/s | 82.9 t/s | — |\n\nQ5_K_M / Q8_0 / BF16 do not fit a single 24GB GPU at usable context.\n\n## Usage\n\n### Text\n\n```bash\nllama-cli -m Qwen3.6-35B-A3B-Q4_K_M.gguf -ngl 99 -c 8192 -p \"Your prompt\"\n```\n\n### Vision (image/video)\n\n```bash\nllama-mtmd-cli -m Qwen3.6-35B-A3B-Q4_K_M.gguf \\\n  --mmproj Qwen3.6-35B-A3B-mmproj-BF16.gguf \\\n  --image path/to/image.jpg -p \"Describe this image\"\n```\n\n## Notes\n\n- Architecture: `Qwen3_5MoeForConditionalGeneration` (`qwen3_5_moe`)\n- Converter: llama.cpp `convert_hf_to_gguf.py` (built-in support)\n- At bs=1 decode, the GPU is kernel-launch bound; prefill or batched serving will show higher utilization\n- For long context (>262K), see base model's YaRN config\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "llama.cpp",
    "qwen3_5_moe",
    "multimodal",
    "vision",
    "moe",
    "image-text-to-text",
    "en",
    "zh",
    "multilingual",
    "base_model:Qwen/Qwen3.6-35B-A3B",
    "base_model:quantized:Qwen/Qwen3.6-35B-A3B",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 541,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-16T15:08:20.000Z",
  "created_at": "2026-04-16T15:03:09.000Z",
  "pipeline_tag": "image-text-to-text",
  "library_name": "gguf"
}

{
  "_id": "69e0fa2da34df4a5ac5f330e",
  "id": "Infatoshi/Qwen3.6-35B-A3B-GGUF",
  "modelId": "Infatoshi/Qwen3.6-35B-A3B-GGUF",
  "sha": "22880479d6dd057b3fab8654afaaf34648fb524e",
  "createdAt": "2026-04-16T15:03:09.000Z",
  "lastModified": "2026-04-16T15:08:20.000Z",
  "author": "Infatoshi",
  "downloads": 541,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "image-text-to-text",
  "library_name": "gguf",
  "siblings_count": 7
}