---

license: apache-2.0

language:

• en
• zh
• hi

tags:

• gguf
• qwen
• qwen3
• llama.cpp
• ollama
• local-llm
• text-generation

pipeline_tag: text-generation

library_name: llama.cpp

base_model:

• Qwen/Qwen3-0.6B

---

Qwen3 Nano GGUF

This repository contains GGUF quantized versions of Qwen3 Nano optimized for local inference using:

• llama.cpp
• Ollama
• LM Studio
• KoboldCpp
• other GGUF-compatible runtimes

GGUF is a file format designed for efficient LLM inference, especially on consumer hardware.

---

Model Information

| Property | Value |

|---|---|

| Model | Qwen3 Nano |

| Format | GGUF |

| Architecture | Transformer decoder-only |

| Quantization | Q4_K_M / Q5_K_M |

| Context Length | 8192 tokens |

| Runtime | llama.cpp compatible |

| License | Apache 2.0 |

---

Available Files

Qwen3-nano-GGUF/
│
├── README.md
├── Qwen3-nano-Q4_K_M.gguf
├── Qwen3-nano-Q5_K_M.gguf
└── config.json

Quantization Details

| File | Description |

|---|---|

| Qwen3-nano-Q4_K_M.gguf | Smaller size, lower RAM usage, faster inference |

| Qwen3-nano-Q5_K_M.gguf | Better quality, slightly higher memory usage |

---

Usage

1. llama.cpp

Install llama.cpp:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

cmake -B build
cmake --build build --config Release

Run the model:

./build/bin/llama-cli \
-m Qwen3-nano-Q4_K_M.gguf \
-p "Explain artificial intelligence"

Interactive chat:

./build/bin/llama-cli \
-m Qwen3-nano-Q4_K_M.gguf \
-i

---

2. Ollama

Create a file named Modelfile:

FROM ./Qwen3-nano-Q4_K_M.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
PARAMETER num_predict 512

SYSTEM """
You are a helpful AI assistant.
Answer clearly and accurately.
"""

Create Ollama model:

ollama create qwen3-nano -f Modelfile

Run:

ollama run qwen3-nano

---

3. LM Studio

1. Download the GGUF file.
2. Open LM Studio.
3. Select:

Qwen3-nano-Q4_K_M.gguf

1. Load and chat locally.

---

Hardware Recommendations

Approximate requirements:

| Quantization | RAM Recommendation |

| ------------ | ------------------ |

| Q4_K_M | 4-6 GB RAM |

| Q5_K_M | 6-8 GB RAM |

A laptop with 8-16 GB RAM can run these models comfortably.

---

Recommended Settings

For general chat:

temperature: 0.7
top_p: 0.9
context: 8192

For coding:

temperature: 0.2-0.5
top_p: 0.9

---

Conversion Information

This GGUF model was converted from the original Hugging Face model using:

llama.cpp convert_hf_to_gguf.py

Conversion workflow:

Hugging Face Model
        |
        v
 llama.cpp Converter
        |
        v
       GGUF
        |
        v
 llama.cpp / Ollama

---

Limitations

• Performance depends on CPU/GPU hardware.
• Quantized models may have slightly lower accuracy than the original FP16 model.
• GGUF models are intended for inference, not training.

---

Credits

Original model:

Qwen3 Nano

https://huggingface.co/Qwen

GGUF conversion:

llama.cpp project

https://github.com/ggerganov/llama.cpp

---

Disclaimer

This repository only provides GGUF conversion files.

Please refer to the original model license and terms of use.