---

license: apache-2.0

base_model: Qwen/Qwen2.5-0.5B-Instruct

tags:

• gguf
• qwen2
• quantized
• llama.cpp
• q4_k_m
• q8_0

pipeline_tag: text-generation

---

Qwen2.5-0.5B-Instruct — GGUF

GGUF quantizations of

Qwen/Qwen2.5-0.5B-Instruct,

for use with llama.cpp and compatible

runtimes (Ollama, LM Studio, llama-cpp-python, etc.).

Two quant levels are provided so you can trade size against quality. For a

model this small, Q8_0 is near-lossless and Q4_K_M is the smallest with

still-solid quality.

| File | Quant | Size | Notes |

|------|-------|------|-------|

| Qwen2.5-0.5B-Instruct-Q4_K_M.gguf | Q4_K_M | ~379 MB | Smallest, good quality |

| Qwen2.5-0.5B-Instruct-Q8_0.gguf | Q8_0 | ~506 MB | Near-lossless |

Usage

llama.cpp

llama-cli -hf andreolf/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 -p "Hello!"

or with a local file:

llama-cli -m Qwen2.5-0.5B-Instruct-Q8_0.gguf \
  -p "Explain what a GGUF file is in one sentence."

Ollama

ollama run hf.co/andreolf/Qwen2.5-0.5B-Instruct-GGUF:Q8_0

How it was made

Converted from the original safetensors to an F16 GGUF with

convert_hf_to_gguf.py, then quantized with llama-quantize (Q4_K_M) and

directly (Q8_0) from llama.cpp. No

additional fine-tuning or modification was applied.

License & attribution

The original model is released by Qwen under Apache-2.0; this quantization

inherits that license. All credit for the underlying model goes to the

Qwen team. This repo only provides a quantized

GGUF conversion for convenience.