---

license: apache-2.0

base_model: ibm-granite/granite-4.0-h-small

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- moe

- apex

- quantized

- granite

- mamba

- hybrid

- llama.cpp

---

Granite-4.0-H-Small — APEX GGUF

MoE-aware, mixed-precision APEX quantizations of

ibm-granite/granite-4.0-h-small

— IBM's hybrid Mamba-2 / Transformer MoE (granitemoehybrid): 40 layers

(36 Mamba-2 + 4 attention), 64-routed + shared-expert MoE, ~32B total / ~9B active,

Apache-2.0. To my knowledge the first APEX quant of a 32B Granite hybrid.

Results & which file to pick

Perplexity on wikitext-2-raw (test, 200×512-token windows), llama-perplexity.

All quants are imatrix-guided (diverse calibration).

| File | Size | BPW | PPL | Δ vs bf16 | Fits (VRAM) |

|------|------|-----|-----|-----------|-------------|

| bf16 (reference) | 61 GB | 16.0 | 5.752 | — | — |

| APEX-i-quality | 21 GB | 5.44 | 5.781 | +0.50% | 24 GB — best quality |

| APEX-i-compact | 16 GB | 4.13 | 5.998 | +4.28% | 24 GB |

| APEX-i-mini | 13 GB | 3.34 | 6.471 | +12.49% | 16 GB — smallest, quality trade |

• i-quality — recommended; near-lossless (+0.5%). Q6_K/Q5_K experts, Q8_0 shared, Q6_K attn/SSM.
• i-compact — Q3_K mid experts; a real but usable quality step (+4.3%) for tighter fits.
• i-mini — IQ2_S mid experts; smallest that stays coherent (16 GB cards), but a **clear

quality drop (+12.5%)** — pick it only when you must fit and can tolerate the hit.

> Note on minimum size: Granite's routed-expert dim is 640 (not 256-divisible), so

> i-quants can't apply to the ffn_down_exps — they fall back to ~Q4_K. That floors the

> practical minimum around ~13 GB (i-mini) and, as a bonus, protects the down projections'

> quality. Going below that (nano/IQ1) buys little here.

Usage (llama.cpp)

llama-cli   -m granite-4.0-h-small-APEX-i-quality.gguf -ngl 999 -p "Hello"
llama-server -m granite-4.0-h-small-APEX-i-compact.gguf -ngl 999 --host 0.0.0.0 --port 8080

Requires a llama.cpp build supporting the granitemoehybrid architecture.

Method

APEX = per-tensor-role, per-layer precision over stock llama-quantize --tensor-type-file.

Granite-H needed its Mamba-2 mixer tensors added to the map (the stock generator omits them):

• Mamba-2 (36 layers): ssm_in, ssm_conv1d, ssm_out at mixer precision; 1-D state

(ssm_a/d/dt/norm) left F32.

• Attention (4 layers): attn_q/k/v/output.
• MoE (all 40): routed ffn__exps on a depth gradient; shared ffn__shexp at Q8_0/Q6_K**.

Baseline: IBM's own f16 GGUF. imatrix from Bartowski calibration_datav3.

Config gen + patcher: configs/, patch_granite_config.py, REPRODUCE.md.

Null result: protecting the Mamba-2 recurrence doesn't help

A hand-roll pinning ssm_in/out/conv1d to Q8_0 (22 GB, PPL 5.779) was within noise of

i-quality — the SSM/recurrence tensors aren't precision-sensitive. Consistent with the

same finding on Kimi-Linear's KDA and Granite-4.0-H-Tiny.

It is shipped here as granite-4.0-h-small-APEX-handroll.gguf for transparency and

reproduction, but there's no reason to use it — it's slightly larger than i-quality

(22 GB vs 21 GB) for no measurable quality gain (PPL 5.779 vs 5.781). Use i-quality.

Attribution & licenses

See LICENSE (Apache-2.0) and NOTICE.

• Base: IBM (@ibm-granite) — granite-4.0-h-small (Apache-2.0)
• Engine: llama.cpp (@ggml-org) (MIT)
• APEX: Ettore Di Giacinto / LocalAI (@mudler) — localai-org/apex-quant (MIT)
• Calibration: Bartowski (@bartowski) — calibration_datav3

Unofficial community quantization; not affiliated with or endorsed by IBM.