steampunque/qwen3.5-9b-mp-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "base_model": "Qwen/Qwen3.5-9B",
    "base_model_relation": "quantized",
    "tags": [
      "Qwen3.5 9B",
      "GGUF",
      "quantized",
      "4-bit",
      "mixed precision"
    ],
    "frontmatter": {
      "license": "apache-2.0",
      "base_model": "Qwen/Qwen3.5-9B",
      "base_model_relation": "quantized",
      "tags": [
        "Qwen3.5 9B",
        "GGUF",
        "quantized",
        "4-bit",
        "mixed precision"
      ]
    },
    "hero_image_url": "",
    "summary": "",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nbase_model: Qwen/Qwen3.5-9B\nbase_model_relation: quantized\ntags:\n- Qwen3.5 9B\n- GGUF\n- quantized\n- 4-bit\n- mixed precision\n---\n\n## Mixed Precision GGUF layer quantization of Qwen3.5-9B by Qwen\n\nOriginal model: https://huggingface.co/Qwen/Qwen3.5-9B\n\nThe mixed precision quant employs different quantization levels on a per layer basis to enable\nboth high performance and small file size at the same time. The quants employed are all K to avoid\nslow CPU or older GPU processing of IQ quants.  For this file the layer quants are as follows:\n```\nQ4_K_L : Q4_K_M + attn_o = q6_k\nQ5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k\nQ6_K_S : Q6_K\n\n   LAYER_TYPES='[\n   [0 ,\"Q5_K_M\"], [1 ,\"Q5_K_S\"], [2 ,\"Q4_K_L\"], [3 ,\"Q4_K_M\"], [4 ,\"Q4_K_S\"], [5 ,\"Q4_K_M\"], [6 ,\"Q4_K_S\"], [7 ,\"Q4_K_M\"],\n   [8 ,\"Q4_K_S\"], [9 ,\"Q4_K_S\"], [10,\"Q4_K_S\"], [11,\"Q4_K_S\"], [12,\"Q4_K_M\"], [13,\"Q4_K_S\"], [14,\"Q4_K_M\"], [15,\"Q4_K_S\"],\n   [16,\"Q4_K_M\"], [17,\"Q4_K_S\"], [18,\"Q4_K_M\"], [19,\"Q4_K_M\"], [20,\"Q4_K_M\"], [21,\"Q4_K_M\"], [22,\"Q4_K_M\"], [23,\"Q4_K_M\"],\n   [24,\"Q4_K_M\"], [25,\"Q4_K_M\"], [26,\"Q4_K_M\"], [27,\"Q4_K_L\"], [28,\"Q5_K_S\"], [29,\"Q5_K_M\"], [30,\"Q5_K_L\"], [31,\"Q6_K_S\"]\n   ]'\n   FLAGS=\"--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high\"\n```\nThe layer quants were optimized for very strong performance across a set of curated reasoning prompts.  The final quant size\nis about 0.6B bigger than Q4_K_M.\n\nComparison:\n\nQuant  |  size   |  PPL  |   Comment\n-------|---------|-------|-----------\nQ4_K_M | 5.6e9  | 7.7   | Q4_K_M with default embedding and output\nQ4_K_H |  6.1e9 | 7.8   | Mixed precision quant with Q6_K embedding Q6_K\n\nUsage:\n\nQwen3.5-9B is a vision capable dense RL model. It can be used together with its multimedia projector layers to process images and text inputs\nand generate text outputs. The mmproj file is made available in this repository.\n\nUpdate 3/18/26: original mmproj had BF16 mmproj tensors. These are still available, unmodified, renamed to *.mmproj.BF16.gguf. New F16 mmproj format is the default to enable working across the widest range of platforms.\n\nSpeculation does not work with the model due to the attention scheme it uses.\nOn a 4070 with all layers and context in VRAM with no vision tower approx performance is:\n\nQ      |   QKV    |  NKV   |  gen tps\n-------|----------|--------|--------\nQ4_K_H | F16      |  166k +|  63  \nQ4_K_H | Q8_0     |  280k +|  64\n\nLong context test (needle in haystack) was tested and passed with fast prompt processing, making large context actually usable with the model.\n\nThe model appears to be trained to decide itself whether to do a think block or not.  When it does a think block it falls into very\nheavy overthinking but does come up with accurate answers.  Over a small set of eval prompts the model did extremely well.  To avoid\nthe overthinking inject think start and think stop tokens first thing after assistant prompt:\n\n```\nTHINK_START=\"<think>\\n\"\nTHINK_STOP=\"\\n</think>\\n\\n\"\n```\n\nIf the model doesnt feel like doing thinking on a given prompt it will automatically do this.  To force the model into\na think block inject a bootstrap think block following the assistant prompt:\n\n```\n\"<think>\\nHere's a thinking process to solve the problem:\"\n```\n\nThe model was found to be highly capable on reasoning tasks when skipping think block, with zero overthinking, just accurate\ndirect deductions to final solutions.  When doing thinking with greedy sampling the model went into an infinite rep loop on one of the test prompts\n(somewhat tricky question it had trouble resolving) but did well on all the remaining prompts.  This is similar behaviour to other\nqwen3 thinkers which have trouble with infinite repeat when using greedy sampling particularly at smaller quant sizes (<10B params)\n\nThe model was tested in vision mode on a couple pretty tough bird ID image and did well, with a very detailed think block and\naccurate final conclusion.\n\nThe model was tested across a small set of code gen prompts and found to be quite intermittent in its ability to generate working code,\nand went into infinite repeat on two of the code prompts where it decided to use a think block when using greedy sampling.\n\nLlama.cpp minimum version to run Qwen3.5-9B should be b8148 and above due to correction of a graph error which\ncauses crashes in both RPC and multiple local GPU setups.  If the model run over RPC it will crash due to an unresolved\nmemory leak in RPC: https://github.com/ggml-org/llama.cpp/issues/19892, temp workaround set GGML_CUDA_DISABLE_GRAPHS=1 on rpc server launch.\n\nBenchmarks:\n\nA full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm\n\n## Download the file from below:\n| Link | Type | Size/e9 B | Notes |\n|------|------|-----------|-------|\n| [Qwen3.5-9B.Q4_K_H.gguf](https://huggingface.co/steampunque/Qwen3.5-9B-MP-GGUF/resolve/main/Qwen3.5-9B.Q4_K_H.gguf) | Q4_K_H | 6.1e9 B | 0.6B bigger than Q4_K_M |\n| [Qwen3.5-9B.mmproj.gguf](https://huggingface.co/steampunque/Qwen3.5-9B-MP-GGUF/resolve/main/Qwen3.5-9B.mmproj.gguf) | F16 | 0.92e9 B | multimedia projector |\n| [Qwen3.5-9B.mmproj.BF16.gguf](https://huggingface.co/steampunque/Qwen3.5-9B-MP-GGUF/resolve/main/Qwen3.5-9B.mmproj.BF16.gguf) | BF16 | 0.92e9 B | multimedia projector |\n \nA discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:\n\nhttps://github.com/ggml-org/llama.cpp/discussions/13040",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "Qwen3.5 9B",
    "GGUF",
    "quantized",
    "4-bit",
    "mixed precision",
    "base_model:Qwen/Qwen3.5-9B",
    "base_model:quantized:Qwen/Qwen3.5-9B",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 5,
  "downloads": 437,
  "gated": false,
  "private": false,
  "last_modified": "2026-03-18T13:08:32.000Z",
  "created_at": "2026-03-02T21:06:51.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "69a5fbeb41e5e425ed8bb17e",
  "id": "steampunque/Qwen3.5-9B-MP-GGUF",
  "modelId": "steampunque/Qwen3.5-9B-MP-GGUF",
  "sha": "a787bfe127c0c48e8e14504bffc7cfe3b78000a7",
  "createdAt": "2026-03-02T21:06:51.000Z",
  "lastModified": "2026-03-18T13:08:32.000Z",
  "author": "steampunque",
  "downloads": 437,
  "likes": 5,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 5
}