Granite Docling 258M GGUF

GGUF conversion of RockMan256/granite-docling-258M, a fine-tuned version of ibm-granite/granite-docling-258M.

Files

| File | Description |

|------|-------------|

| granite-docling-258M-bf16.gguf (317 MB) | LLM weights in BF16 |

| mmproj-model-f16.gguf (182 MB) | Vision projector (mmproj) in F16 |

Usage with llama.cpp

# Build llama.cpp (if not already built)
cd llama.cpp && make -j

# Run with multimodal (image + text) inference
./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  --mmproj mmproj-model-f16.gguf \
  --chat-template chatml

Example prompts

# Describe an image
echo "Describe this image in detail." | ./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  --mmproj mmproj-model-f16.gguf \
  -p /path/to/image.png

# Text-only inference (no vision)
echo "What is the capital of France?" | ./llama-mtmd-cli \
  -m granite-docling-258M-bf16.gguf \
  -p - --chat-template chatml

Conversion Details

• LLM: Converted from user's fine-tuned RockMan256/granite-docling-258M (original weights, not remote)
• mmproj: Converted from ibm-granite/granite-docling-258M (vision encoder/connector unchanged from base)
• Quantization: BF16 (LLM), F16 (mmproj)

License

Apache 2.0 (same as base model)