---

base_model: Qwen/Qwen3.5-9B

license: apache-2.0

library_name: llama.cpp

pipeline_tag: text-generation

language:

• en
• zh

tags:

• gguf
• llama.cpp
• qwen3.5
• quantization
• traditional-chinese

---

Qwen3.5-9B GGUF Quant Lab

Reproducible llama.cpp GGUF quantization ladder for Qwen/Qwen3.5-9B, evaluated on English and Traditional Chinese perplexity, TMMLU+, throughput, and VRAM.

This repository contains the text-only language model. It does not include a vision projector (mmproj) and cannot provide the original model's image-input capability.

Recommended default

IQ4_XS-imatrix is the measured sweet spot on the RTX 4090 test system: 4.840 GiB, English/Traditional-Chinese PPL increases of +1.50%/+1.95% from F16, and a TMMLU+ change of -1.0 percentage point. Choose Q4_K_M-imatrix when you prefer slightly stronger PPL retention at a larger 5.243 GiB size.

Quantization and evaluation

Hardware: RTX 4090 24 GiB, Intel i7-13700, WSL2 Ubuntu, CUDA 13.1, llama.cpp b10031. Performance is hardware-specific. PPL is a next-token likelihood measure, not an instruction-following or safety score. TMMLU+ may have appeared in the base model's training data; here it is used only for relative quantization degradation because every quantization receives the identical fixed 500-question sample.

| Quant | GiB | English PPL | 繁中 PPL | TMMLU+ | pp512 tok/s | tg128 tok/s |

| --- | ---: | ---: | ---: | ---: | ---: | ---: |

| F16 | 16.690 | 8.0111 | 10.0015 | 68.8% | 7589 | 212 |

| Q8_0 | 8.873 | 8.0190 | 9.9952 | 69.2% | 9038 | 85 |

| Q6_K | 6.854 | 8.0160 | 10.0306 | 69.4% | 7545 | 99 |

| Q5_K_M | 6.024 | 8.0591 | 10.1387 | 68.2% | 8704 | 114 |

| Q4_K_M | 5.243 | 8.1110 | 10.2830 | 68.4% | 8881 | 123 |

| Q3_K_M | 4.306 | 9.0446 | 11.1233 | 63.0% | 8140 | 141 |

| Q2_K | 3.564 | 100.8741 | 44.6410 | 28.6% | 7412 | 159 |

| Q4_K_M-imatrix | 5.243 | 8.0865 | 10.1508 | 68.2% | 8777 | 101 |

| IQ4_XS-imatrix | 4.840 | 8.1311 | 10.1969 | 67.8% | 9539 | 135 |

Full methodology, paired bootstrap CIs, Q2_K regression examples, figures, and reproducibility metadata are in EVAL_REPORT.md.

llama.cpp

./llama-cli -m Qwen3.5-9B-IQ4_XS-imatrix.gguf -ngl 99 --jinja --reasoning off -p "請用繁體中文介紹 GGUF 量化。"

Ollama

Download a GGUF file and the included Modelfile, then run:

ollama create qwen35-9b-quant-lab -f Modelfile
ollama run qwen35-9b-quant-lab

The supplied Modelfile points at Qwen3.5-9B-IQ4_XS-imatrix.gguf. Change FROM to select another local GGUF.

LM Studio

Download one .gguf file, open LM Studio's local model picker, and select the file. This is a Qwen3.5 text-only model; keep the embedded GGUF chat template enabled and disable reasoning if you want the same evaluation-style behavior.

Attribution and license

Base model: Qwen/Qwen3.5-9B at revision c202236235762e1c871ad0ccb60c8ee5ba337b9a. The original model and these quantized derivative weights are distributed under Apache-2.0; see LICENSE. Project scripts and documentation are MIT-licensed. This repository was staged for steven0226/Qwen3.5-9B-GGUF-Quant-Lab.