arelath/Qwen3-0.6B-nanoquant-GGUF overview
Experiment 28: Qwen/Qwen3 0.6B quality benchmark Status: completed Model: Qwen/Qwen3 0.6B Revision: c1899de289a04d12100db370d81485cdf75e47ca Candidate run: D:\…
Runs locally from ~375.2 MB disk (4 GB VRAM class GPUs with llama.cpp / guIDE).
Repository Files & Downloads
| File | Type | Quantization | Size | Link |
|---|---|---|---|---|
| qwen3-0-6b-nanoquant.gguf | GGUF | GGUF | 375.2 MB | Download |
Model Details
| Model ID | arelath/Qwen3-0.6B-nanoquant-GGUF |
|---|---|
| Author | arelath |
| Pipeline | text-generation |
| License | apache-2.0 |
| Base model | Qwen/Qwen3-0.6B |
| Last modified | 2026-07-25T06:11:03.000Z |
Model README
---
base_model: Qwen/Qwen3-0.6B
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3-0.6B/blob/main/LICENSE
pipeline_tag: text-generation
tags:
- gguf
- nanoquant
- quantized
base_model_relation: quantized
---
Experiment 28: Qwen/Qwen3-0.6B quality benchmark
- Status:
completed - Model:
Qwen/Qwen3-0.6B - Revision:
c1899de289a04d12100db370d81485cdf75e47ca - Candidate run:
D:\dev\research\NanoQuantRewrite\evidence\028\028-compress-and-benchmark-qwen3-0-6b - Backend:
llama.cpp - Wall time: 336.10 seconds
completed means all evaluators returned finite metrics; it is not a BF16-quality acceptance gate.
Protocol
- WikiText-2: 64 samples × 128 tokens, batch 8
- WikiText token hash:
sha256:e85e3ae8da01294229ac7dfc651f5d44f74eadaff974ad80a5a19779b58fbe0f - Tasks: piqa, arc_easy, arc_challenge, hellaswag, winogrande, boolq; first 200 rows, batch 4
- Tokenizer hash:
sha256:af4e928cfa630ef78563f4800a751d568307a7cd4d4f5fd5919de27b0fbfdbc4
Quality results
| Benchmark | Metric | BF16 | NanoQuant | Delta | Ratio |
| --- | --- | ---: | ---: | ---: | ---: |
| WikiText-2 | perplexity ↓ | 55.166982 | 292.263889 | +237.096907 (+429.78%) | 5.2978x |
| piqa | acc_norm ↑ | 0.6900 | 0.5500 | -0.1400 | 0.7971x |
| arc_easy | acc_norm ↑ | 0.5650 | 0.3450 | -0.2200 | 0.6106x |
| arc_challenge | acc_norm ↑ | 0.3400 | 0.2600 | -0.0800 | 0.7647x |
| hellaswag | acc_norm ↑ | 0.5100 | 0.3950 | -0.1150 | 0.7745x |
| winogrande | acc ↑ | 0.6050 | 0.5150 | -0.0900 | 0.8512x |
| boolq | acc ↑ | 0.6550 | 0.6250 | -0.0300 | 0.9542x |
Deployment storage
| Deployable representation | Bytes | Storage savings vs BF16 |
| --- | ---: | ---: |
| BF16 checkpoint tensors | 1,503,300,328 | — |
| NanoQuant GGUF | 393,472,384 | 73.83% |
The packed NanoQuant quantized-layer payload is 56,725,568 bytes. The complete GGUF is the comparable deployable artifact and also includes embeddings, the output projection, norms, and metadata.
Quality execution
The BF16 reference uses Transformers. The NanoQuant candidate is evaluated only from the exported GGUF by the pinned llama.cpp runtime; no reconstructed dense or factorized PyTorch candidate is loaded.
| Runtime | Elapsed seconds | Peak dedicated GPU bytes | Peak host bytes |
| --- | ---: | ---: | ---: |
| BF16 Transformers | 28.94 | 5,205,131,264 | 2,352,574,464 |
| NanoQuant llama.cpp GGUF | 306.33 | 1,096,601,600 | 997,515,264 |
- GGUF:
D:\dev\research\NanoQuantRewrite\Results\028\qwen3-0-6b-nanoquant.gguf - GGUF SHA-256:
sha256:b0ffbcc384324f5500a040c78fc649197925fdccf30139b9335cdda8b3cf6011 - llama.cpp commit:
2927569f318311aded148d76be6b1109664066f5
Provenance
- Experiment config hash:
sha256:8a0ebce1b4e0ccbaeef076b5a4cd0910e9f137984f66f2a39b47642416daeeda - Launcher:
experiments/028-compress-and-benchmark-qwen3-0-6b.py - Candidate identity:
{"config_hash":"sha256:6f5ef817ff06cfa29ee046f08800b7179bb574efc0ad914dce1a717e8d94fa29","model_hash":"sha256:be8e62ef3fcac5fb56dc193b416d77ca862ec1594958ada22bd5c002f6a065de","plan_hash":"sha256-3fe3842dd0ccde5b34961fd71302e681ecadf3b0653d9e9f49131aadfe0f2d51"} - Global tuning:
None
Run arelath/Qwen3-0.6B-nanoquant-GGUF with guIDE
Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.
Source: Hugging Face · Compare models