arelath/Qwen3-8B-nanoquant-GGUF overview
Experiment 29: Qwen/Qwen3 8B quality benchmark Status: completed Model: Qwen/Qwen3 8B Revision: b968826d9c46dd6066d109eabc6255188de91218 Candidate run: /worksp…
Runs locally from ~2.06 GB disk (4 GB VRAM class GPUs with llama.cpp / guIDE).
Repository Files & Downloads
| File | Type | Quantization | Size | Link |
|---|---|---|---|---|
| qwen3-8b-nanoquant.gguf | GGUF | GGUF | 2.06 GB | Download |
Model Details
| Model ID | arelath/Qwen3-8B-nanoquant-GGUF |
|---|---|
| Author | arelath |
| Pipeline | text-generation |
| License | apache-2.0 |
| Base model | Qwen/Qwen3-8B |
| Last modified | 2026-07-25T14:07:56.000Z |
Model README
---
base_model: Qwen/Qwen3-8B
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3-8B/blob/main/LICENSE
pipeline_tag: text-generation
tags:
- gguf
- nanoquant
- quantized
base_model_relation: quantized
---
Experiment 29: Qwen/Qwen3-8B quality benchmark
- Status:
completed - Model:
Qwen/Qwen3-8B - Revision:
b968826d9c46dd6066d109eabc6255188de91218 - Candidate run:
/workspace/NanoQuant/evidence/029/029-compress-and-benchmark-qwen3-8b - Backend:
llama.cpp - Wall time: 1769.80 seconds
completed means all evaluators returned finite metrics; it is not a BF16-quality acceptance gate.
Protocol
- WikiText-2: 64 samples × 128 tokens, batch 8
- WikiText token hash:
sha256:e85e3ae8da01294229ac7dfc651f5d44f74eadaff974ad80a5a19779b58fbe0f - Tasks: piqa, arc_easy, arc_challenge, hellaswag, winogrande, boolq; first 200 rows, batch 4
- Tokenizer hash:
sha256:af4e928cfa630ef78563f4800a751d568307a7cd4d4f5fd5919de27b0fbfdbc4
Quality results
| Benchmark | Metric | BF16 | NanoQuant | Delta | Ratio |
| --- | --- | ---: | ---: | ---: | ---: |
| WikiText-2 | perplexity ↓ | 23.110475 | 40.599726 | +17.489251 (+75.68%) | 1.7568x |
| piqa | acc_norm ↑ | 0.8000 | 0.5350 | -0.2650 | 0.6687x |
| arc_easy | acc_norm ↑ | 0.8050 | 0.4000 | -0.4050 | 0.4969x |
| arc_challenge | acc_norm ↑ | 0.5350 | 0.3400 | -0.1950 | 0.6355x |
| hellaswag | acc_norm ↑ | 0.6350 | 0.4050 | -0.2300 | 0.6378x |
| winogrande | acc ↑ | 0.7050 | 0.5700 | -0.1350 | 0.8085x |
| boolq | acc ↑ | 0.8550 | 0.6600 | -0.1950 | 0.7719x |
Deployment storage
| Deployable representation | Bytes | Storage savings vs BF16 |
| --- | ---: | ---: |
| BF16 checkpoint tensors | 16,381,516,776 | — |
| NanoQuant GGUF | 2,214,297,824 | 86.48% |
The packed NanoQuant quantized-layer payload is 884,765,160 bytes. The complete GGUF is the comparable deployable artifact and also includes embeddings, the output projection, norms, and metadata.
Quality execution
The BF16 reference uses Transformers. The NanoQuant candidate is evaluated only from the exported GGUF by the pinned llama.cpp runtime; no reconstructed dense or factorized PyTorch candidate is loaded.
| Runtime | Elapsed seconds | Peak dedicated GPU bytes | Peak host bytes |
| --- | ---: | ---: | ---: |
| BF16 Transformers | 34.98 | 18,886,950,912 | 4,983,508,992 |
| NanoQuant llama.cpp GGUF | 1731.46 | unavailable | 2,550,489,088 |
- GGUF:
/workspace/NanoQuant/Results/029/qwen3-8b-nanoquant.gguf - GGUF SHA-256:
sha256:75d3f4f150195e0c6a7ccd2ce3372f919534fb7ec609f6b9ba52eb71af194253 - llama.cpp commit:
2927569f318311aded148d76be6b1109664066f5
Provenance
- Experiment config hash:
sha256:993eca2ef6c345d01926e4677e9cf9620c001fa7e61f1ffa207eeaad1e1803cb - Launcher:
experiments/029-compress-and-benchmark-qwen3-8b.py - Candidate identity:
{"config_hash":"sha256:9356ea7a7e46c6dc5286ce574c110add86e4fd77b9f0c92dc32f4646888f9dd9","model_hash":"sha256:db9a057024370dc56a8bba76b9f274692c2094615dc4da1497ce830d14707bc6","plan_hash":"sha256-bc87a0d973d42d9bc247fd72660dc76c590ffe29fbf918f1aa8db56e87c4dd8a"} - Global tuning:
None
Run arelath/Qwen3-8B-nanoquant-GGUF with guIDE
Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.
Source: Hugging Face · Compare models