---

title: granite-4.2-3b-GGUF

library_name: llama.cpp

model_type: quantized

base_model: ibm-granite/granite-4.2-3b

datasets: []

tags: [gguf, quantized, text-generation, granite, granite-4.2, reasoning, thinking, tool-calling, llama.cpp]

---

Granite-4.2-3B GGUF (llama.cpp)




Quantized GGUF files for Granite-4.2-3B, optimized for deployment with llama.cpp. These are the exact quantized versions of the Granite-4.2-3B model, packaged as GGUF format for efficient inference.

Model Overview

Granite-4.2-3B is IBM's compact reasoning model in the Granite 4.2 family. Despite its small parameter count of 3B, it delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought reasoning. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.

Key Capabilities

• Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems
• Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model
• Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls
• 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows
• Apache 2.0 Licensed: Fully open for commercial and research use

Available Quantizations

| File | Size | Quality | Recommended Use |

|------|------|---------|-----------------|

| granite-4.2-3b-BF16.gguf | 6.82 GB | Full precision source. Every quant below is cut from this file. | Original model, no quantization |

| granite-4.2-3b-Q8_0.gguf | 3.63 GB | Effectively lossless. Use when disk and RAM are not the constraint. | Highest quality, less compression |

| granite-4.2-3b-Q6_K.gguf | 2.80 GB | Near-lossless; the last stop before quality becomes measurable. | Balanced quality/size |

| granite-4.2-3b-Q5_K_M.gguf | 2.43 GB | Very good quality, noticeably smaller than Q6_K. | Good trade-off |

| granite-4.2-3b-Q5_K_S.gguf | 2.38 GB | Slightly smaller than Q5_K_M for a slight quality cost. | Smaller footprint |

| granite-4.2-3b-Q5_1.gguf | 2.58 GB | Legacy. Prefer Q5_K_M. | Historical compatibility |

| granite-4.2-3b-Q5_0.gguf | 2.38 GB | Legacy. Prefer Q5_K_M. | Historical compatibility |

| granite-4.2-3b-Q4_K_M.gguf | 2.09 GB | The usual default. Best quality-per-byte for most people. | Default choice |

| granite-4.2-3b-Q4_K_S.gguf | 2.00 GB | A little smaller than Q4_K_M, a little worse. | Smaller footprint |

| granite-4.2-3b-IQ4_NL.gguf | 1.99 GB | Non-linear 4-bit; good on hardware without fast K-quant kernels. | Specialized hardware |

| granite-4.2-3b-IQ4_XS.gguf | 1.89 GB | Best sub-4.5bpw option; usually beats Q4_K_S at a smaller size. | Size-critical |

| granite-4.2-3b-Q4_1.gguf | 2.18 GB | Legacy. Prefer Q4_K_M. | Historical compatibility |

| granite-4.2-3b-Q4_0.gguf | 1.98 GB | Legacy round-to-nearest. Prefer Q4_K_M unless a runtime needs this. | Historical compatibility |

| granite-4.2-3b-Q3_K_L.gguf | 1.84 GB | Small, with real quality loss. Usable when RAM is tight. | Tight RAM constraints |

| granite-4.2-3b-Q3_K_M.gguf | 1.71 GB | Smaller again; noticeable degradation. | Memory-constrained |

| granite-4.2-3b-IQ3_M.gguf | 1.61 GB | Strong at ~3.7bpw, clearly better than Q3_K_M. | Quality-conscious sizing |

| granite-4.2-3b-IQ3_S.gguf | 1.57 GB | Slightly smaller than IQ3_M. | Compact version |

| granite-4.2-3b-Q3_K_S.gguf | 1.56 GB | Aggressive. Prefer IQ3_M at a similar size. | Maximum compression |

| granite-4.2-3b-IQ3_XS.gguf | 1.50 GB | Aggressive but coherent. | Extreme compression |

| granite-4.2-3b-IQ3_XXS.gguf | 1.39 GB | Very aggressive; the last coherent step down. | Extreme compression |

| granite-4.2-3b-Q2_K.gguf | 1.36 GB | Very small, heavily degraded. For experimentation. | Experimental only |

| granite-4.2-3b-IQ2_M.gguf | 1.26 GB | The smallest size most people find usable. | Memory-constrained |

| granite-4.2-3b-Q2_K_S.gguf | 1.27 GB | Smaller than Q2_K, at a further quality cost. | Even smaller |

| granite-4.2-3b-IQ2_S.gguf | 1.18 GB | Below the usual usability line. | Extreme compression |

| granite-4.2-3b-IQ2_XS.gguf | 1.11 GB | Experimental. | Experimental only |

| granite-4.2-3b-IQ2_XXS.gguf | 1.02 GB | Experimental. | Experimental only |

| granite-4.2-3b-Q2_0.gguf | 1.16 GB | Extreme, group-64. Included for completeness. | Historical compatibility |

| granite-4.2-3b-IQ1_M.gguf | 0.92 GB | Extreme. Expect substantial degradation. | Maximum compression |

| granite-4.2-3b-IQ1_S.gguf | 0.87 GB | Extreme. Expect substantial degradation. | Maximum compression |

| granite-4.2-3b-Q1_0.gguf | 0.65 GB | Extreme. Included for completeness. | Historical compatibility |

All files are cut from the BF16 source (6.82 GB). Each quant is independently uploaded and deleted immediately after successful upload to minimize peak disk usage.

Model Architecture

Granite-4.2-3B is built on a decoder-only dense transformer architecture:

• Attention: Grouped Query Attention (GQA) with 40 attention heads and 8 KV heads
• Position Embedding: Rotary Position Embedding (RoPE) with θ = 10,000,000
• Feed-Forward: MLP with SwiGLU activation (hidden size 8192)
• Normalization: RMSNorm (ε = 1e-5)
• Embeddings: Separate input/output embeddings (not tied)
• Precision: bfloat16 (source)

Inference

Generation Parameters

> Important: Use temperature=1.0 and top_p=0.95 across all tasks and serving backends, including general chat, reasoning, and tool calling.

| Parameter | Value | Notes |

|:----------|:------|:------|

| temperature | 1.0 | Required for all modes |

| top_p | 0.95 | Nucleus sampling threshold |

| max_new_tokens | 8192 | Thinking mode (increase for complex reasoning) |

| max_new_tokens | 2048 | Non-thinking mode |

| do_sample | True | Required when temperature > 0 |

Thinking Modes

| Mode | Template Parameters | Behavior |

|:-----|:-------------------|:---------|

| Thinking (default) | enable_thinking=True | Full chain-of-thought reasoning inside <think>...</think> |

| Non-thinking | enable_thinking=False | Direct answer with no reasoning overhead |

| Low-effort | enable_thinking=True, low_effort=True | Brief reasoning for simpler queries |

Serving with llama.cpp

Basic Usage

# Download a quantization (recommended: Q4_K_M)
huggingface-cli download NANI-Nithin/granite-4.2-3b-GGUF granite-4.2-3b-Q4_K_M.gguf --local-dir .

# Run inference
llama-cli -m granite-4.2-3b-Q4_K_M.gguf -p "Hello, how are you?"

Or use the Hugging Face Hub integration:

# Serve with llama-server
llama-server -hf NANI-Nithin/granite-4.2-3b-GGUF:Q4_K_M

# Run with model path
llama-cli -hf NANI-Nithin/granite-4.2-3b-GGUF:Q4_K_M -p "Hello"

API Usage

# Start the server
llama-server -hf NANI-Nithin/granite-4.2-3b-GGUF:Q4_K_M

# Query with curl
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "granite-4.2-3b-Q4_K_M",
    "messages": [{"role": "user", "content": "Explain quantum computing in simple terms"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "max_tokens": 8192
  }'

Quick Start Example

# Download the model
huggingface-cli download NANI-Nithin/granite-4.2-3b-GGUF granite-4.2-3b-Q4_K_M.gguf

# Run inference
./llama-cli -m granite-4.2-3b-Q4_K_M.gguf -p "What is the Riemann hypothesis?"

Technical Details

• Source Model: ibm-granite/granite-4.2-3b
• Author: Granite Team, IBM
• License: Apache 2.0
• Languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, Chinese
• Architecture: GraniteForCausalLM (decoder-only dense transformer)
• Context Length: 131K tokens (natively supports 128K, extended to 512K)
• Parameters: 3B
• Created: August 25, 2026

Usage Notes

1. Disk Space: Download one quantization at a time. Each file ranges from 0.65 GB (Q1_0) to 6.82 GB (BF16 source).
2. Memory Requirements: Varies by quantization. Q4_K_M requires ~2 GB VRAM, Q8_0 requires ~4 GB VRAM.
3. Performance: All files are optimized for llama.cpp with appropriate kernel selections for each quantization.
4. Thinking Mode: Enable enable_thinking=True to get chain-of-thought reasoning. Set to False for faster, direct answers.

Model Card Information

This GGUF repo contains quantized versions of Granite-4.2-3B, including evaluation results and comprehensive serving instructions. The quantization was performed using llama.cpp's quantization pipeline, preserving the model's reasoning capabilities while reducing size for deployment.

For the full source model documentation, including detailed training methodology, evaluation benchmarks, and advanced usage, refer to the source repo: ibm-granite/granite-4.2-3b.

---

Note: These files are optimized for llama.cpp and are not compatible with vLLM, SGLang, or the Transformers library in their current format. For those frameworks, use the source model ibm-granite/granite-4.2-3b.