---

license: apache-2.0

base_model: Qwen/Qwen3-4B-Instruct-2507

tags:

- qwen

- qwen3

- gguf

- quantized

- mixed-precision

- bitclass

pipeline_tag: text-generation

library_name: gguf

---

Qwen3-4B-Instruct-2507 — BitClass Mixed-Precision GGUF

Mixed-precision GGUF quantizations of Qwen3-4B-Instruct-2507 using learned per-tensor quantization profiles. Each tensor group receives the precision level that minimizes quality loss for its importance — more bits where they matter, fewer where they don't.

Seven precision levels from ultra-compact (3.0 bpw) to high quality (4.5 bpw). Low-BPW models use IQ types for better quality-per-bit; higher levels use KQ types for faster inference.

Models

| File | Target BPW | Size | PPL ↓ | Family |

|---|---|---|---|---|

| Qwen3-4B-Instruct-2507-MX-3.0bpw.gguf | 3.0 | 1.72 GB | 3.073 | IQ |

| Qwen3-4B-Instruct-2507-MX-3.2bpw.gguf | 3.2 | 1.79 GB | 3.034 | IQ |

| Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf | 3.4 | 1.88 GB | 2.971 | IQ |

| Qwen3-4B-Instruct-2507-MX-3.6bpw.gguf | 3.6 | 2.02 GB | 2.936 | KQ |

| Qwen3-4B-Instruct-2507-MX-3.8bpw.gguf | 3.8 | 2.03 GB | 2.951 | KQ |

| Qwen3-4B-Instruct-2507-MX-4.0bpw.gguf | 4.0 | 2.01 GB | 2.928 | KQ |

| Qwen3-4B-Instruct-2507-MX-4.5bpw.gguf | 4.5 | 2.47 GB | 2.932 | KQ |

> Target BPW is the planner's per-tensor bit budget (and the filename label). The actual

> whole-file BPW runs ~0.3–0.4 higher, because output/embedding tensors are kept at higher

> precision and GGUF carries metadata overhead — see the Size column for the real footprint.

Recommended: MX-3.4bpw for the best quality-to-size ratio. MX-3.0bpw for maximum compression. MX-4.0bpw for the highest quality in this ladder.

How It Compares

| Model | BPW | Size | PPL ↓ | Source |

|---|---|---|---|---|

| ByteShape KQ 3.34 | 3.34 | 1.69 GB | 3.121 | byteshape |

| ★ Ours MX-3.0 | 3.0 | 1.72 GB | 3.073 | This repo |

| ★ Ours MX-3.2 | 3.2 | 1.79 GB | 3.034 | This repo |

| Unsloth Q3_K_S | 3.75 | 1.75 GB | 3.007 | unsloth |

| ★ Ours MX-3.4 | 3.4 | 1.88 GB | 2.971 | This repo |

| ★ Ours MX-3.6 | 3.6 | 2.02 GB | 2.936 | This repo |

| ★ Ours MX-4.0 | 4.0 | 2.01 GB | 2.928 | This repo |

| Unsloth Q4_K_M | 4.97 | 2.32 GB | 2.956 | unsloth |

| ★ Ours MX-4.5 | 4.5 | 2.47 GB | 2.932 | This repo |

All models benchmarked in the same session on identical hardware (NVIDIA GB10 ATOM, GPU) for fair comparison.

Key Results

• Lower perplexity than ByteShape KQ 3.34: MX-3.0 (PPL 3.073) vs 3.121 — 1.5% better, at comparable size (1.72 vs 1.69 GB)
• Lower perplexity than Unsloth Q3_K_S: MX-3.4 (PPL 2.971) vs 3.007 — 1.2% better, at 1.88 vs 1.75 GB
• Lower perplexity than Unsloth Q4_K_M: MX-4.5 (PPL 2.932) vs 2.956 — 0.8% better, at 2.47 vs 2.32 GB

Running with Ollama

# Pick any precision level
ollama run hf.co/sh111111111111111/Qwen3-4B-Instruct-2507-BitClass-GGUF:Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf

Running with llama.cpp

# Chat
llama-cli -m Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf -cnv

# Server (OpenAI-compatible API)
llama-server -m Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf --port 8080

# Benchmark
llama-perplexity -m Qwen3-4B-Instruct-2507-MX-3.4bpw.gguf -f your_eval_data.txt

Benchmarking Details

All benchmarks run with llama.cpp (commit 406f4e3) on NVIDIA GB10 ATOM GPU with full offload (-ngl 999). Perplexity measured via llama-perplexity on a held-out evaluation set (20 chunks, 512 context). Throughput via llama-bench (512 prompt / 128 generation tokens). All models — ours, Unsloth, ByteShape — benchmarked in the same session.

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.

License

Apache 2.0, inherited from Qwen3-4B-Instruct-2507.

Acknowledgments

• Base model by Qwen Team
• Importance matrix data from Unsloth and Bartowski
• Quantization infrastructure built on llama.cpp