---

language: [en, zh]

license: apache-2.0

library_name: gguf

base_model: Qwen/Qwen3-4B-Instruct-2507

tags: [quantized, gguf, mixed-precision, bitclass, qwen3]

pipeline_tag: text-generation

---

Qwen3-4B-Instruct-2507 — BitClass2 Mixed-Precision GGUF

Mixed-precision GGUF quantizations of Qwen3-4B-Instruct-2507

using Hessian-informed per-tensor bit allocation. Each tensor group receives

the precision level that minimizes quality loss for its measured sensitivity —

more bits where they matter, fewer where they don't.

Available Quantizations

| File | BPW | Size | PPL ↓ | tok/s | Use Case |

|---|---|---|---|---|---|

| Qwen3-4B-Instruct-2507-Q8_0.gguf | 8.5 | 4.28 GB | 2.651 | 11.4 | Near-lossless reference |

| Qwen3-4B-Instruct-2507-Q6_K.gguf | 5.8 | 2.93 GB | 2.888 | 13.6 | High quality, moderate size |

| Qwen3-4B-Instruct-2507-Q5_K_M.gguf | 5.2 | 2.60 GB | 2.971 | 14.3 | Balanced quality and size |

| Qwen3-4B-Instruct-2507-Q4_K_M.gguf | 4.7 | 2.35 GB | 2.978 | 14.1 | Best quality-to-size ratio |

| Qwen3-4B-Instruct-2507-Q3_K_S.gguf | 3.2 | 1.62 GB | 3.214 | 18.9 | Maximum compression |

Recommended: Q4_K_M for the best quality-to-size ratio (PPL 2.978 at just 2.35 GB).

Q3_K_S for maximum compression. Q6_K for high quality.

How It Works

Standard quantization applies one precision level uniformly across all tensors.

BitClass2 uses Hessian-based sensitivity analysis (H_diag = mean(X²) per layer)

to identify which tensors lose the most quality when quantized, then solves an

LP-optimal knapsack allocation: minimize Σ(sensitivity × quantization_error)

subject to total size ≤ target. Sensitive tensors get higher precision,

insensitive ones get lower precision, at the same total file size.

Within each suffix group, the fractional BPW planner further varies types

per-layer using blended imatrix + Hessian scores, so late attention layers

(most sensitive) get higher precision than middle layers (least sensitive).

Key Sensitivity Findings (Qwen3-4B)

• Late attention layers (29-35) are most sensitive — blk.34 k/v score 1.0
• down_proj is the most sensitive MLP tensor — projects back to residual stream
• gate_proj/up_proj are least sensitive — safe to quantize aggressively
• K > V for attention weight sensitivity — k_proj averages 0.66 vs v_proj 0.50

Usage

# Download
huggingface-cli download sh111111111111111/Qwen3-4B-Instruct-2507-BitClass2-GGUF \
    Qwen3-4B-Instruct-2507-Q4_K_M.gguf --local-dir .

# Chat with llama.cpp
llama-cli -m Qwen3-4B-Instruct-2507-Q4_K_M.gguf -cnv

# Serve via API
llama-server -m Qwen3-4B-Instruct-2507-Q4_K_M.gguf --port 8080

# Ollama
ollama run hf.co/sh111111111111111/Qwen3-4B-Instruct-2507-BitClass2-GGUF:Qwen3-4B-Instruct-2507-Q4_K_M.gguf

Benchmark Details

All benchmarks run on NVIDIA GB10 ATOM (128GB unified memory, aarch64).

llama.cpp commit 406f4e3. PPL via llama-perplexity (2 chunks, 851 context).

tok/s via llama-bench (tg128, ngl=999).

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp.

License

Apache 2.0, inherited from Qwen3-4B-Instruct-2507.