---

library_name: gguf

base_model: Qwen/Qwen2.5-32B-Instruct

tags:

• gguf
• llama.cpp
• quantized
• q4_k_m

---

⚡ Quantized with QuantizeLab

This model was quantized to GGUF format using QuantizeLab, the fastest SaaS to quantize models under 32B parameters. Join hundreds of developers downloading our optimized quants!

• Fast Conversion: Direct upload to Hugging Face.
• Optimized Sizes: Support for all major GGUF bit-rates.
• Hardware Free: No local GPU required for quantization.

👉 Try QuantizeLab Now

Qwen2.5-32B-Instruct-GGUF (Q4_K_M)

Q4_K_M GGUF quantization of Qwen/Qwen2.5-32B-Instruct,

produced with llama.cpp by

Quantizelab.dev.

| | |

| --- | --- |

| File | model-Q4_K_M.gguf |

| Quantization | Q4_K_M |

| Size on disk | 19.85 GB |

| Base model | Qwen/Qwen2.5-32B-Instruct |

Run it (full GPU offload)

GGUF defaults to CPU. To get GPU speed you must offload every layer — a

single layer left on the CPU takes generation from ~25 tok/s to ~3 tok/s.

-ngl 999 simply means "offload all of them".

# llama.cpp
llama-cli -hf thecodehaider/Qwen2.5-32B-Instruct-GGUF:model-Q4_K_M.gguf -ngl 999 -c 4096 -p "Hello"

# local file
llama-cli -m model-Q4_K_M.gguf -ngl 999 -c 4096 -cnv

# OpenAI-compatible server
llama-server -m model-Q4_K_M.gguf -ngl 999 -c 4096 --port 8080

# Ollama
ollama run hf.co/thecodehaider/Qwen2.5-32B-Instruct-GGUF:Q4_K_M

# llama-cpp-python
from llama_cpp import Llama
llm = Llama(model_path="model-Q4_K_M.gguf", n_gpu_layers=-1, n_ctx=4096)
print(llm("Hello", max_tokens=128)["choices"][0]["text"])

Will it fit your GPU?

Weights plus ~1.2 GB of KV-cache and compute overhead at a 4k context.

| GPU | VRAM | Fits fully offloaded? | Headroom for context |

| --- | ---- | --------------------- | -------------------- |

| NVIDIA T4 / RTX 4060 | 16 GB | No | offload partially (-ngl lower) or use CPU |

| RTX 3090 / 4090 / A10 | 24 GB | Tight | ~2.9 GB (cap context ~2048) |

| A100 40GB | 40 GB | Yes | ~18.9 GB (4k+ context) |

If a row says No, lower -ngl until it fits, or run on CPU (GGUF works

either way — it is just slower).

Notes

• Q4_K_M is the recommended balance of size and quality; Q8_0 and above

will not fully offload to a 16 GB card for models past ~8B.

• Reduce -c (context) first when you hit out-of-memory: the KV cache grows

linearly with context length.