---

license: apache-2.0

base_model: ibm-granite/granite-embedding-97m-multilingual-r2

tags:

- gguf

- llama.cpp

- embeddings

- granite

- modernbert

- edge

---

granite-embedding-97m-multilingual-r2-GGUF

F16 GGUF conversion of ibm-granite/granite-embedding-97m-multilingual-r2 for local serving with llama.cpp. Converted and independently verified by ATF (Agent Taskflow) for edge-local embedding serving via atf-serve.

This is a format conversion only — no weights were modified, retrained, or fine-tuned. All model weights are © IBM, licensed Apache-2.0 (same as the base model). This repository is not affiliated with or endorsed by IBM.

Why this exists

IBM does not publish a GGUF for this model. This repo documents its own build end-to-end — source checksum, conversion command, and independent correctness verification — rather than asking you to trust an unverified re-hosted binary.

Conversion details

• Source: ibm-granite/granite-embedding-97m-multilingual-r2, model.safetensors (bf16)
• Tool: llama.cpp built from source at commit 11924d4c17abc27383376a1ac6a24fa3e36c1c0c (2026-08-02). This model uses its own purpose-trained 180,000-token multilingual tokenizer — a distinct tokenizer from the 311m sibling model, registered upstream as granite-embed-multi-97m (a different pre-tokenizer type than the 311m variant's granite-embed-multi-311m/GEMMA4). Neither registration is recognized by llama.cpp release b9204 or earlier; a current build is required both to convert and to serve this model.
• Command:

```

python3 convert_hf_to_gguf.py <model-dir> \

--outfile granite-embedding-97m-multilingual-r2-f16.gguf \

--outtype f16

```

• Output: F16, 384-dim, 206,403,072 bytes.

Verification (independent, not vendor-claimed)

Embedded the same test sentence through both this GGUF (via llama-server --embedding --pooling cls) and the original HF model (via sentence-transformers, loaded directly from the source safetensors), then computed cosine similarity between the two output vectors. Verified on both CPU and GPU (CUDA) inference paths.

| Check | Result |

|---|---|

| Output dimension | 384 (matches source hidden_size) |

| Cosine similarity vs. HF reference pipeline (CPU) | 0.999980 |

| Cosine similarity vs. HF reference pipeline (GPU/CUDA) | 0.999980 |

| Required pooling mode | cls (matches source classifier_pooling: "cls" in config.json; mean pooling is not correct for this model) |

| CPU footprint (process RSS, mmap'd) | ~299 MB |

| GPU VRAM (full offload, -ngl 99) | ~354 MB |

| Latency (single request, CPU / GPU) | ~23 ms / ~10 ms |

See the sibling granite-embedding-311m-multilingual-r2-GGUF for the larger variant.

Usage

llama-server --model granite-embedding-97m-multilingual-r2-f16.gguf \
  --embedding --pooling cls --port 8089

Requires a llama.cpp build that includes granite-embed-multi-97m tokenizer support (see Conversion details above — current upstream master has it; check your pinned release tag if serving fails with an unknown pre-tokenizer type error).

curl http://127.0.0.1:8089/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "your text here", "model": "granite-embedding-97m"}'

---

Converted by ATF — agent orchestration with edge-local model serving.