---

language: [en, zh]

license: apache-2.0

library_name: gguf

base_model: Qwen/Qwen3.5-9B

tags: [quantized, gguf, mixed-precision, bitclass, qwen3]

pipeline_tag: text-generation

---

Qwen3.5-9B — BitClass2 Mixed-Precision GGUF

Mixed-precision GGUF quantizations of Qwen3.5-9B

using Hessian-informed per-tensor bit allocation. Each tensor group receives

the precision level that minimizes quality loss for its measured sensitivity.

Available Quantizations

| File | BPW | Size | PPL ↓ | tok/s | Use Case |

|---|---|---|---|---|---|

| Qwen3.5-9B-Q8_0.gguf | 8.5 | 9.53 GB | 1.728 | 6.5 | Near-lossless reference |

| Qwen3.5-9B-Q6_K.gguf | 5.6 | 6.30 GB | 1.863 | 8.3 | High quality |

| Qwen3.5-9B-Q5_K_M.gguf | 5.1 | 5.70 GB | 1.865 | 8.8 | Balanced quality and size |

| Qwen3.5-9B-Q4_K_M.gguf | 4.7 | 5.21 GB | 1.875 | 9.2 | Best quality-to-size ratio |

| Qwen3.5-9B-Q3_K_S.gguf | 3.3 | 3.66 GB | 2.001 | 11.5 | Maximum compression |

Recommended: Q4_K_M — nearly matches Q6_K quality (PPL 1.875 vs 1.863) at 17% less size.

How It Compares

| Model | BPW | Size | PPL ↓ | Source |

|---|---|---|---|---|

| ByteShape IQ3_S 3.00bpw | 3.0 | 3.37 GB | 2.069 | byteshape |

| ★ Ours Q3_K_S | 3.3 | 3.66 GB | 2.001 | This repo |

| ★ Ours Q4_K_M | 4.7 | 5.21 GB | 1.875 | This repo |

| ★ Ours Q5_K_M | 5.1 | 5.70 GB | 1.865 | This repo |

Our Q3_K_S beats ByteShape's 3.00bpw 9B on perplexity (2.001 vs 2.069 — 3.3% better), at a larger file (3.66 vs 3.37 GB). ByteShape's higher-BPW rows reach lower PPL.

The PPL curve is remarkably flat from Q6_K to Q4_K_M: going from 6.30 GB down to

5.21 GB (saving ~1.1 GB) only costs 0.012 PPL. This is the mixed-precision allocation

working — gate_proj/up_proj drop to Q4_K while down_proj and attention stay at Q6_K.

Key Sensitivity Findings (Qwen3.5-9B)

The Hessian sensitivity pattern for 9B is fundamentally different from 4B:

• blk.3 (early layer) is most sensitive — score 1.0 for k/v. On 4B it was blk.34 (late layer).
• Sensitivity peaks at both ends AND middle: blk.3 (1.0), blk.7 (0.78), blk.23 (0.78), blk.27 (0.86), blk.31 (0.87)
• ffn_down at blk.4-5 is near-zero sensitivity (0.0003) — safe for aggressive quantization
• This confirms: model-specific Hessian data matters. You cannot assume late layers are always most sensitive.

How It Works

1. Hessian sensitivity — compute H_diag = mean(X²) per layer on calibration data
2. LP-optimal allocation — solve knapsack: minimize Σ(sensitivity × quant_error) subject to size ≤ target
3. Per-layer variation — within each suffix group, vary types by layer using imatrix + Hessian blend
4. GGUF export — llama.cpp --tensor-type-file for per-tensor overrides

Usage

huggingface-cli download sh111111111111111/Qwen3.5-9B-BitClass2-GGUF \
    Qwen3.5-9B-Q4_K_M.gguf --local-dir .

llama-cli -m Qwen3.5-9B-Q4_K_M.gguf -cnv
llama-server -m Qwen3.5-9B-Q4_K_M.gguf --port 8080

Benchmark Details

NVIDIA GB10 ATOM (128GB unified memory, aarch64). llama.cpp commit 406f4e3.

PPL via llama-perplexity (2 chunks, 851 context). tok/s via llama-bench (tg128, ngl=999).

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.

License

Apache 2.0, inherited from Qwen3.5-9B.