---

license: apache-2.0

library_name: gguf

base_model: ibm-granite/granite-switch-4.1-3b-preview

tags:

• language
• granite-switch
• granite-4.1
• gguf

---

Granite Switch 4.1 3B Preview - GGUF (bf16)

GGUF conversion of ibm-granite/granite-switch-4.1-3b-preview, for use with llama.cpp.

• Quantization: bf16 (no quantization loss, same precision as the original checkpoint)
• Converted with: convert_hf_to_gguf.py from llama.cpp (master)
• Original model card / adapters / usage: see ibm-granite/granite-switch-4.1-3b-preview

Files

| File | Description |

|---|---|

| granite-switch-4.1-3b-preview-bf16.gguf | bf16 GGUF, includes base model weights and all 12 embedded LoRA adapters |

Basic usage

llama-cli -m granite-switch-4.1-3b-preview-bf16.gguf -p "Hello"

With no control token, the model runs as the plain base chat model.

Adapter activation

Granite Switch embeds 12 adapters in one checkpoint. An in-graph router selects

which adapter is active by detecting a control token in the prompt. The selection

is per request (one adapter per generation) — the control token routes the whole

sequence to a single adapter slot, then is substituted out before embedding so it

does not corrupt the text.

To activate an adapter you place its control token in the prompt at the position

the model was trained for. Placement differs by adapter type:

• LoRA adapters — the token goes at the very start, replacing the leading

<|start_of_role|>. Adapters: citations, hallucination_detection,

context-attribution.

• aLoRA adapters — the token is spliced right before the final

assistant<|end_of_role|> generation prompt (replacing its <|start_of_role|>).

Adapters: query_rewrite, query_clarification, answerability,

factuality-detection, policy-guardrails, factuality-correction,

guardian-core, uncertainty, requirement-check.

| Adapter | Type | Control token | Output |

|---|---|---|---|

| citations | lora | <\|citations\|> | citation spans |

| hallucination_detection | lora | <\|hallucination_detection\|> | [{"r","f":faithful/partial/unfaithful/NA,"e"}] |

| context-attribution | lora | <\|context-attribution\|> | attribution spans |

| query_rewrite | alora | <\|query_rewrite\|> | rewritten query |

| query_clarification | alora | <\|query_clarification\|> | clarification or CLEAR |

| answerability | alora | <\|answerability\|> | answerable / unanswerable |

| factuality-detection | alora | <\|factuality-detection\|> | yes / no |

| policy-guardrails | alora | <\|policy-guardrails\|> | Yes / No / Ambiguous |

| factuality-correction | alora | <\|factuality-correction\|> | corrected text |

| guardian-core | alora | <\|guardian-core\|> | yes / no (risk) |

| uncertainty | alora | <\|uncertainty\|> | 0..9 (certainty) |

| requirement-check | alora | <\|requirement-check\|> | yes / no |

Example: hallucination_detection (LoRA — token at start)

<|hallucination_detection|>user<|end_of_role|>Tell me about the moon.<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>The moon is made of green cheese.<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>

Produces the adapter's structured JSON, e.g.

[{"r": 0, "f": "unfaithful", "e": "..."}] instead of a chat reply.

Example: answerability (aLoRA — token before the assistant prompt)

<|start_of_role|>system<|end_of_role|>You are a helpful assistant with access to the following documents...
<documents>
{"doc_id": "1", "text": "The square root of 4 is 2."}
</documents>
...<|end_of_text|>
<|start_of_role|>user<|end_of_role|>What is the square root of 4?<|end_of_text|>
<|answerability|>assistant<|end_of_role|>

Produces answerable (or unanswerable for a question the documents don't cover).

The exact formats above are what the model's own chat template renders. The

reliable way to build them is to render the template with adapter_name set —

`tokenizer.apply_chat_template(messages, documents=..., adapter_name="answerability",

add_generation_prompt=True, tokenize=False)` — rather than hand-constructing them.

Running the adapters with Ollama

Because adapter selection lives in the ggml graph and expects the control token

already present in the prompt, the cleanest path through Ollama is a raw request

where you supply the fully-rendered prompt yourself:

ollama create granite-switch-4.1-3b-preview -f Modelfile   # FROM the bf16 GGUF

curl http://localhost:11434/api/generate -d '{
  "model": "granite-switch-4.1-3b-preview",
  "raw": true,
  "prompt": "<|start_of_role|>user<|end_of_role|><|hallucination_detection|>The moon is made of green cheese.<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>",
  "stream": false,
  "options": {"temperature": 0}
}'

raw: true bypasses Ollama's chat template so the control token reaches the model

verbatim. (The normal /api/chat path works too if you embed the token in the

message content, but it cannot reproduce the aLoRA boundary placement for you —

raw mode is the faithful option.)

A current Ollama build already includes granite-switch support, so **no patched

Ollama is required** — you can also drive the adapters with

Mellea against stock Ollama, letting Mellea render the

template and place the control tokens.

License

Apache 2.0, inherited from the base model.