steampunque/qwen3-coder-next-mp-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "base_model": "Qwen/Qwen3-Coder-Next",
    "base_model_relation": "quantized",
    "tags": [
      "Qwen",
      "Qwen3 Coder Next",
      "GGUF",
      "quantized",
      "4-bit",
      "mixed precision"
    ],
    "frontmatter": {
      "license": "apache-2.0",
      "base_model": "Qwen/Qwen3-Coder-Next",
      "base_model_relation": "quantized",
      "tags": [
        "Qwen",
        "Qwen3 Coder Next",
        "GGUF",
        "quantized",
        "4-bit",
        "mixed precision"
      ]
    },
    "hero_image_url": "",
    "summary": "",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nbase_model: Qwen/Qwen3-Coder-Next\nbase_model_relation: quantized\ntags:\n- Qwen\n- Qwen3 Coder Next\n- GGUF\n- quantized\n- 4-bit\n- mixed precision\n---\n\n## Mixed Precision GGUF layer quantization of Qwen3-Coder-Next by Qwen\n\nOriginal model: https://huggingface.co/Qwen/Qwen3-Coder-Next\n\nThe hybrid quant employs different quantization levels on a per layer basis to increase\nflexibility of trading off performance vs file size.  Less parameter bits are used at deep layers\nand more bits at cortex layers to simulultaneously optimize quantized size and model performance.\nFor this file the layer quants are as follows:\n```\nQ4_K_L : Q4_K_M + attn_o = q6_k\nQ5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k\nQ6_K_S : Q6_K\n\n   LAYER_TYPES='[\n   [0 ,\"Q5_K_S\"],[1 ,\"Q4_K_L\"],[2 ,\"Q4_K_M\"],[3 ,\"Q4_K_S\"],[4 ,\"Q4_K_S\"],[5 ,\"Q4_K_S\"],[6 ,\"Q4_K_S\"],[7 ,\"Q4_K_S\"],\n   [8 ,\"Q4_K_S\"],[9 ,\"Q4_K_S\"],[10,\"Q4_K_S\"],[11,\"Q4_K_S\"],[12,\"Q4_K_S\"],[13,\"Q4_K_S\"],[14,\"Q4_K_S\"],[15,\"Q4_K_S\"],\n   [16,\"Q4_K_S\"],[17,\"Q4_K_S\"],[18,\"Q4_K_S\"],[19,\"Q4_K_S\"],[20,\"Q4_K_M\"],[21,\"Q4_K_S\"],[22,\"Q4_K_M\"],[23,\"Q4_K_S\"],\n   [24,\"Q4_K_M\"],[25,\"Q4_K_M\"],[26,\"Q4_K_M\"],[27,\"Q4_K_M\"],[28,\"Q4_K_M\"],[29,\"Q4_K_M\"],[30,\"Q4_K_M\"],[31,\"Q4_K_M\"],\n   [32,\"Q4_K_M\"],[33,\"Q4_K_M\"],[34,\"Q4_K_M\"],[35,\"Q4_K_M\"],[36,\"Q4_K_M\"],[37,\"Q4_K_M\"],[38,\"Q4_K_M\"],[39,\"Q4_K_M\"],\n   [40,\"Q4_K_M\"],[41,\"Q4_K_M\"],[42,\"Q4_K_M\"],[43,\"Q4_K_L\"],[44,\"Q5_K_S\"],[45,\"Q5_K_M\"],[46,\"Q5_K_L\"],[47,\"Q6_K_S\"]\n   ]'\n   FLAGS=\"--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high\"\n```\nThe layer quants were optimized for 100% success across a small set of code generation test prompts while sized for\noperation on machines with 48G CPU RAM and one consumer grade GPU (8G VRAM or higher).  The model was found to be very\nsensitive to quantization degradation negatively impacting its ability to generate working code.\n\nComparison:\n\nQuant    |  size   |  PPL  |   Comment\n---------|---------|-------|-----------\nQ4_K_M   | 48.5e9  |  7.64 | default embed and output\nQ4_K_H   | 48.3e9  |  7.65 | Q6_K embed Q6_K output\n\nUsage:\n\nThis is a 80B parameter coding optimized moe model with 3B activated parameters. It can be efficiently run by offloading expert\ntensors to CPU via -ot exps=CPU to open up very large context space on even low VRAM GPUs.  The smaller size of the optimally\nquantized parameters will give an effective boost in CPU processing speed due to reducing the memory BW needed to repeatedly copy them\nfrom main memory to SIMD regs.\n\nThe model cannot be speculated due to use of some kind of recurrent attention scheme which prohibits it.  In this\nparticular case the limitation is not severe since experts will have to be run on CPU for most setups which makes speculation\nimpractical anyway.  The model was sized at ~48G and should run on a 48G RAM machine.  For unknown reasons it appears necessary to\noffload all experts to CPU; if some expert layers are left on GPU gen rate is cut in half as of llama.cpp b7972.  What this most likely\nmeans is some backend op the model needs cant run on cuda so it kicks the GPU experts over to CPU dynamically every token, slowing the whole\nsystem down.  If and when this problems gets fixed in llama.cpp it should be feasilbe to do partial CPU offload and gain some tps speedup over\nfull CPU expert offload.\n\nRough performance metrics on a 9900k (128G RAM) and 4070 (12G VRAM)\n\nCPU EXP OFFLOAD | QKV  | Context size| gen rate  |  ot config\n----------------|------|-------------|-----------|------------\nall         | F16  |   256K      |  20.1     | OT=\"-ot exps=CPU -ngl 99\"\n4-48        | F16  |   256K      |  8        | OT=\"-ot blk\\\\.[4-9]\\|1[0-9]\\|2[0-9]\\|3[0-9]\\|4[0-7].*exps=CPU -ngl 99\"\n7-48        | F16  |   128K      |  10       | OT=\"-ot blk\\\\.[7-9]\\|1[0-9]\\|2[0-9]\\|3[0-9]\\|4[0-7].*exps=CPU -ngl 99\"\n7-48        | Q8_0 |   256K      |  10       | OT=\"-ot blk\\\\.[7-9]\\|1[0-9]\\|2[0-9]\\|3[0-9]\\|4[0-7].*exps=CPU -ngl 99\"\n9-48        | Q8_0 |   128K      |  10       | OT=\"-ot blk\\\\.[9-9]\\|1[0-9]\\|2[0-9]\\|3[0-9]\\|4[0-7].*exps=CPU -ngl 99\"\n\nHigh context performance appears to work verified against a needle in haystack prompt.  However, prompt processing is\ntoo slow to be practically used on very large prompts without a much stronger CPU or full GPU offload of the model.\n\nBenchmarks:\n\nCode evals for the model will eventually be given here:  https://huggingface.co/spaces/steampunque/benchlm.\n\n## Download the file from below:\n| Link | Type | Size/e9 B | Notes |\n|------|------|-----------|-------|\n| [Qwen3-Coder-Next.Q4_K_H.gguf](https://huggingface.co/steampunque/Qwen3-Coder-Next-MP-GGUF/resolve/main/Qwen3-Coder-Next.Q4_K_H.gguf) | Q4_K_H | 48.39 B | ~Q4_K_M size |\n\nA discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:\n\nhttps://github.com/ggml-org/llama.cpp/discussions/13040",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "Qwen",
    "Qwen3 Coder Next",
    "GGUF",
    "quantized",
    "4-bit",
    "mixed precision",
    "base_model:Qwen/Qwen3-Coder-Next",
    "base_model:quantized:Qwen/Qwen3-Coder-Next",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 3,
  "downloads": 88,
  "gated": false,
  "private": false,
  "last_modified": "2026-03-01T02:46:34.000Z",
  "created_at": "2026-02-09T18:41:21.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "698a2a518416a324f617e744",
  "id": "steampunque/Qwen3-Coder-Next-MP-GGUF",
  "modelId": "steampunque/Qwen3-Coder-Next-MP-GGUF",
  "sha": "e1b67905345e40e39d4dfb6ea4572b94907768b3",
  "createdAt": "2026-02-09T18:41:21.000Z",
  "lastModified": "2026-03-01T02:46:34.000Z",
  "author": "steampunque",
  "downloads": 88,
  "likes": 3,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 3
}