---

language:

• en

license: apache-2.0

tags:

• gguf
• llama-cpp
• granite
• ibm
• reasoning
• tool-use
• quantized
• conversational
• on-device

pipeline_tag: text-generation

base_model: ibm-granite/granite-4.2-8b

---

Granite 4.2 8B · GGUF Quantizations

Ready-to-run GGUF quantizations of IBM Granite 4.2 8B for local inference with llama.cpp, Ollama, LM Studio, Open WebUI, and other GGUF-compatible runtimes.

This repository provides a complete range of quantization formats, from near-lossless BF16 and Q8 models to highly compressed IQ variants for memory-constrained devices.

About the Base Model

Granite 4.2 8B is IBM's dense reasoning language model designed for agentic workflows, coding, tool use, structured outputs, and enterprise-grade reasoning.

Key Properties

| Property | Value |

|----------|--------|

| Base Model | ibm-granite/granite-4.2-8b |

| Parameters | ~8B |

| Architecture | GraniteForCausalLM |

| Context Window | 128K tokens |

| Reasoning Mode | Switchable thinking / non-thinking |

| Tool Calling | Supported |

| License | Apache 2.0 |

| Format | GGUF |

Features

• 🧠 Native reasoning with optional <think> mode
• 🛠️ Tool calling and agent workflows
• 💻 Strong coding and software engineering capabilities
• 📊 Structured JSON generation
• ⚡ Compatible with llama.cpp and GGUF ecosystems
• 📱 Efficient local deployment on consumer hardware
• 🌍 Multilingual support

Recommended Quantizations

| Quant | Recommendation |

|---------|---------------|

| Q6_K | Highest quality without full precision |

| Q5_K_M | Excellent balance of quality and VRAM |

| Q4_K_M | Default recommendation for most users |

| IQ4_XS | Compact deployment with good quality |

| Q3_K_M | Low-memory environments |

| Q2_K | Maximum compression |

Usage with llama.cpp

./llama-cli \
  -m granite-4.2-8b-Q4_K_M.gguf \
  -c 8192 \
  -p "Explain transformer attention."

Prompt Format

<|im_start|>system
You are a helpful assistant.
<|im_end|>

<|im_start|>user
Explain transformer attention.
<|im_end|>

<|im_start|>assistant

Hardware Guidance

| Quant | Approximate RAM / VRAM |

|---------|----------------------|

| Q2_K | 4 GB |

| Q3_K_M | 5 GB |

| Q4_K_M | 6 GB |

| Q5_K_M | 7 GB |

| Q6_K | 9 GB |

| Q8_0 | 11 GB |

| BF16 | 18+ GB |

Actual memory consumption depends on context length, batch size, and runtime configuration.

Intended Use

This model is suitable for:

• Conversational AI
• Coding assistants
• Software engineering agents
• Retrieval-Augmented Generation (RAG)
• Local AI deployments
• Research and experimentation
• Tool-calling workflows

Limitations

• Quantization may slightly reduce model quality compared to the original BF16 checkpoint.
• Smaller quantizations prioritize memory efficiency over accuracy.
• Model outputs may contain inaccuracies or hallucinations.
• Performance depends heavily on the selected quantization level and hardware.

License

This repository distributes quantized versions of IBM Granite 4.2 8B.

The original model is licensed under Apache 2.0. Please refer to the upstream model card for complete licensing and usage information.

Acknowledgements

• IBM Granite Team for the original Granite 4.2 model.
• llama.cpp contributors for GGUF support and quantization tooling.
• Hugging Face for model hosting and distribution.

Citation

If you use these quantizations in research or production, please cite the original Granite 4.2 model from IBM.