---

license: mit

library_name: gguf

tags:

- gguf

- llama.cpp

- phi-4

- phi-4-mini

- microsoft

- instruct

- quantized

- local-inference

pipeline_tag: text-generation

---

Phi-4-mini-instruct GGUF

GGUF versions of Microsoft Phi-4-mini-instruct prepared

for efficient local inference.

This repository contains an F16 GGUF version and a Q4_K_M

quantized version.

Available Files

| File | Format | Approx. Size |

|---|---|---:|

| Phi-4-mini-instruct-F16.gguf | F16 | 7.2 GB |

| Phi-4-mini-instruct-Q4_K_M.gguf | Q4_K_M | 2.4 GB |

Recommended Version

Phi-4-mini-instruct-Q4_K_M.gguf is recommended for most

consumer hardware and local inference setups.

It provides a substantially smaller memory footprint than F16

while retaining the advantages of the GGUF format.

Original Model

The original model is Microsoft's official Phi-4-mini-instruct.

Original model:

https://huggingface.co/microsoft/Phi-4-mini-instruct

Please refer to Microsoft's original model repository for

technical details, intended use, limitations, and licensing.

Quantization

The Q4_K_M file was generated from the F16 GGUF model using

llama.cpp quantization tools.

Quantization format:

Q4_K_M

llama.cpp

Basic usage:

llama-cli -m Phi-4-mini-instruct-Q4_K_M.gguf

GPU offloading:

llama-cli -m Phi-4-mini-instruct-Q4_K_M.gguf -ngl 99

Local server:

llama-server -m Phi-4-mini-instruct-Q4_K_M.gguf -ngl 99

Compatible Software

• llama.cpp
• llama-cpp-python
• LM Studio
• Jan
• Other GGUF-compatible runtimes

Model Variants

F16

Higher precision with substantially larger memory requirements.

Q4_K_M

Smaller model size and lower memory requirements, making it

more suitable for local deployment on consumer hardware.

Disclaimer

This is a community GGUF conversion and quantization.

It is not an official Microsoft repository.

License

Please follow the licensing terms of the original Microsoft

Phi-4-mini-instruct model.

SHA256

Checksums can be generated with:

sha256sum *.gguf