---

base_model: Qwen/Qwen2.5-72B-Instruct

library_name: gguf

pipeline_tag: text-generation

tags: [gguf, llama.cpp, ollama, lm-studio, quantized, lecore]

---

Qwen2.5-72B-Instruct-lecore-GGUF

GGUF builds of Qwen/Qwen2.5-72B-Instruct for llama.cpp, Ollama and LM Studio.

These are companions to the MLX export staccs/Qwen2.5-72B-Instruct-lecore-mlx, which has the measured perplexity numbers.

| File | Size | What it is |

|---|---|---|

| Qwen2.5-72B-Instruct-Q4_K_M.gguf | 47.42 GB | stock llama.cpp Q4_K_M |

| Qwen2.5-72B-Instruct-lecore-mix.gguf | 31.00 GB | leCore per-tensor mix: each linear layer gets the GGUF K-quant matching the bits leCore measured for it (2→Q2_K, 3→Q3_K, 4→Q4_K, 8→Q8_0); embeddings and output follow the MLX export |

The lecore-mix file is quantized from the original bf16 weights with llama.cpp K-quants. The bit width of each tensor follows leCore's measured (KL-damage) allocation, so its numbers are close to, but not identical to, the MLX export.

Some tensor widths are not multiples of 256, so llama.cpp may fall back to a compatible type for those tensors.

Usage

Ollama: download a file, then

printf 'FROM ./Qwen2.5-72B-Instruct-Q4_K_M.gguf
' > Modelfile && ollama create qwen2.5-72b-instruct-lecore-gguf -f Modelfile && ollama run qwen2.5-72b-instruct-lecore-gguf

or pull straight from the Hub with ollama run hf.co/staccs/Qwen2.5-72B-Instruct-lecore-GGUF:Q4_K_M

LM Studio: search for staccs/Qwen2.5-72B-Instruct-lecore-GGUF in the model browser and pick a file. On a 36 GB Mac, the lecore-mix file is the one that fits.

llama.cpp

llama-cli -hf staccs/Qwen2.5-72B-Instruct-lecore-GGUF -p "Hello" -n 256

License

Base model: Qwen/Qwen2.5-72B-Instruct, released under the Qwen License Agreement (https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE). This quantized derivative is distributed under the same agreement, including its commercial-use terms. Only quantization was done here: no retraining.

Benchmarks (lm-eval-harness 0.4.9, run on the shipped weights dequantized to bf16 on GPU)

MMLU 5-shot on the first 10 questions of each of the 57 subjects; GSM8K 5-shot on the first 100 test problems (greedy, 320 new tokens, no chat template). These are subsets, so expect about ±2–4 points of sampling noise.

| Model | MMLU (5-shot) | GSM8K strict | GSM8K flexible |

|---|---|---|---|

| leCore mix (MLX weights; GGUF lecore-mix uses the same per-tensor bits) | 82.1% | 83.0% | 75.0% |

| stock 3-bit affine, group 64 (simulated) | 84.0% | 92.0% | 76.0% |

Only measured numbers are reported. No downstream-task benchmarks were run. Short-context perplexity does not tell you everything about quality, so check outputs for your own use case.