---

license: apache-2.0

base_model: Qwen/Qwen3-4B-Instruct-2507

tags:

- qwen

- qwen3

- gguf

- quantized

- mixed-precision

pipeline_tag: text-generation

library_name: gguf

---

Qwen3-4B-Instruct-2507 — BitClass MX-2 (Mixed-Precision, 4.00 bpw)

A GGUF-quantized version of Qwen3-4B-Instruct-2507 using BitClass, our learned mixed-precision quantization.

This is the MX-2 (quality) variant at 4.00 bits per weight, optimized for quality while remaining compact. For a smaller/faster variant, see MX-1 (3.54 bpw).

Model

| File | Bits/Weight | Size | Perplexity ↓ | Throughput (GPU) | Throughput (CPU) |

| --- | --- | --- | --- | --- | --- |

| Qwen3-4B-Instruct-2507-Q3_K_S-4.00bpw.gguf | 4.00 | 2.01 GB | 2.979 | 79.3 tok/s | 10.5 tok/s |

Benchmark Results

All models evaluated using lm-evaluation-harness v0.4.11 (0-shot) on identical hardware. Higher is better for all metrics.

| Model | BPW | Size | ARC-C ↑ | GSM8K ↑ | IFEval ↑ | TruthfulQA ↑ | Avg |

| --- | --- | --- | --- | --- | --- | --- | --- |

| Unsloth Q5_K_M | 5.75 | 2.69 GB | 58.79 | 72.55 | 57.49 | 62.49 | 62.83 |

| Unsloth Q4_K_M | 4.97 | 2.33 GB | 57.76 | 66.87 | 55.27 | 60.75 | 60.16 |

| Ours MX-2 | 4.00 | 2.01 GB | 57.42 | 51.40 | 53.60 | 60.21 | 55.66 |

| Ours MX-1 | 3.54 | 1.78 GB | 53.75 | 53.60 | 50.46 | 60.14 | 54.49 |

| ByteShape KQ 3.34 | 3.34 | 1.69 GB | 55.72 | 45.56 | 51.20 | 58.41 | 52.72 |

| Unsloth Q3_K_S | 3.75 | 1.76 GB | 55.89 | 41.24 | 52.13 | 60.10 | 52.34 |

ARC-C: acc_norm, GSM8K: exact_match (flexible-extract), IFEval: prompt_level_strict_acc, TruthfulQA: acc (mc2). All values ×100.

MX-2 at a glance:

• +3.32 average over Unsloth Q3_K_S — GSM8K gap is +10.16 points
• +2.94 average over ByteShape KQ 3.34bpw — wins on every task
• ARC-C (57.42) nearly matches Q4_K_M (57.76) at smaller size (2.01 vs 2.33 GB)
• TruthfulQA (60.21) on par with larger models

Perplexity Comparison

!Tradeoff Chart

| Model | BPW | Size | PPL ↓ | Source |

| --- | --- | --- | --- | --- |

| Unsloth Q5_K_M | 5.75 | 2.69 GB | 2.907 | unsloth |

| Ours MX-2 | 4.00 | 2.01 GB | 2.979 | This repo |

| Unsloth Q3_K_S | 3.75 | 1.76 GB | 3.007 | unsloth |

| Unsloth Q4_K_M | 4.97 | 2.33 GB | 2.956 | unsloth |

| ByteShape KQ 3.34 | 3.34 | 1.69 GB | 3.175 | byteshape |

• Within ~0.8% of Unsloth Q4_K_M perplexity (2.979 vs 2.956) at 14% smaller size (2.01 vs 2.33 GB)
• Lower perplexity than Unsloth Q3_K_S (2.979 vs 3.007)
• Beats ByteShape KQ 3.34bpw by 6.2% (2.979 vs 3.175)

!Precision Loss Chart

Quantization Labels

The filename label Q3_K_S indicates the base quantization type. The actual model uses a mix of quantization types across tensor groups, with an average effective bits per weight of 4.00.

Running with Ollama

ollama run hf.co/sh111111111111111/Qwen3-4B-Instruct-2507-BitClass-MX2-GGUF:Qwen3-4B-Instruct-2507-Q3_K_S-4.00bpw.gguf

Running with llama.cpp

# Chat
llama-cli -m Qwen3-4B-Instruct-2507-Q3_K_S-4.00bpw.gguf -cnv

# Server (OpenAI-compatible API)
llama-server -m Qwen3-4B-Instruct-2507-Q3_K_S-4.00bpw.gguf --port 8080

Evaluation Details

• Perplexity & Throughput: llama.cpp b8514, measured on both NVIDIA GB10 GPU (-ngl 999) and CPU
• Task benchmarks: lm-evaluation-harness v0.4.11, 0-shot, via llama-cpp-python with logits_all=True
• All models benchmarked in the same session on identical hardware for fair comparison

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.

License

Apache 2.0, inherited from Qwen3-4B-Instruct-2507.

Acknowledgments

• Base model by Qwen Team
• Importance matrix data from Unsloth
• Quantization infrastructure built on llama.cpp