---

license: apache-2.0

base_model: ibm-granite/granite-4.0-h-tiny

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- moe

- apex

- quantized

- granite

- mamba

- hybrid

- llama.cpp

---

Granite-4.0-H-Tiny — APEX GGUF

MoE-aware, mixed-precision APEX quantization of

ibm-granite/granite-4.0-h-tiny

— IBM's hybrid Mamba-2 / Transformer MoE (granitemoehybrid): 40 layers = 36

Mamba-2 + 4 attention, a 64-routed + shared-expert MoE FFN on every layer, ~7B

total / ~1B active, Apache-2.0.

To my knowledge this is the first APEX quant of a Granite hybrid Mamba-2/MoE.

APEX assigns precision per tensor role and per layer; here that meant teaching the

recipe about the Mamba-2 mixer tensors the stock generator doesn't know (see Method).

Results

Perplexity on wikitext-2-raw (test, 200×512-token windows), llama-perplexity.

| File | Size | BPW | PPL | Δ vs bf16 |

|------|------|-----|-----|-----------|

| bf16 (reference) | 13 GB | 16.0 | 8.868 | — |

| APEX i-quality (ssm@Q6) | 4.4 GB | 5.40 | 8.901 | +0.38% |

| APEX i-quality — torch-imatrix | 4.4 GB | 5.40 | 8.864 | −0.04% |

| APEX hand-roll (ssm@Q8) | 4.5 GB | 5.54 | 8.913 | +0.51% |

The torch-imatrix row is the same APEX i-quality recipe, quantized with an

importance matrix computed independently in PyTorch (see [*The torch-imatrix

variant*](#the-torch-imatrix-variant)) instead of llama-imatrix. Its PPL (8.864)

is inside the ±0.11 error bar of both bf16 and the standard i-quality — i.e. the

two imatrices are interchangeable in quality.

Within 0.38% of full-precision perplexity at ~3× smaller, and it runs comfortably

on modest hardware (~14 GB bf16 → 4.4 GB). Coherent, ~110 tok/s on a single GPU.

Built with a diverse imatrix (Bartowski calibration_datav3), full expert coverage.

Usage (llama.cpp)

llama-cli   -m granite-4.0-h-tiny-APEX-i-quality.gguf -ngl 999 -p "Hello"
llama-server -m granite-4.0-h-tiny-APEX-i-quality.gguf -ngl 999 --host 0.0.0.0 --port 8080

Requires a llama.cpp build supporting the granitemoehybrid (a.k.a. granitehybrid)

architecture.

Method

APEX is a bit-allocation recipe over stock llama-quantize --tensor-type-file.

Granite-H needed the Mamba-2 mixer tensors added to the map, which the stock

APEX generator omits:

• Mamba-2 (36 layers): ssm_in, ssm_conv1d, ssm_out at mixer precision

(Q6_K); 1-D state (ssm_a, ssm_d, ssm_dt, ssm_norm) left F32.

• Attention (4 layers): standard attn_q/k/v/output (Q6_K).
• MoE (all 40 layers): routed ffn_*_exps on a layer-depth precision gradient

(Q6_K/Q5_K/IQ4_XS), *shared experts ffn__shexp at Q8_0** (always active → protect),

router left high. Routed intermediate dim 512 is 256-divisible → no IQ4_NL workaround.

Config generation + patcher: configs/, patch_granite_config.py, REPRODUCE.md.

Baseline: IBM's own bf16 GGUF.

Which file to pick, and a null result

Use i-quality — smaller and better. The hand-roll tier (also shipped) pins

the Mamba-2 recurrence tensors (ssm_in/out/conv1d) to Q8_0 to test whether protecting

the linear-recurrence state helps: it doesn't — it's larger and slightly worse

(4.5 GB, PPL 8.913 vs 8.901). Same null result seen on Kimi-Linear's KDA, so across two

hybrid architectures the SSM/recurrence tensors just aren't precision-sensitive. The

hand-roll is included to document the experiment; i-quality is the recommended tier.

The torch-imatrix variant

granite-4.0-h-tiny-APEX-i-quality-torch.gguf (+ granite-4.0-h-tiny-torch.imatrix)

is a companion build that answers one question: **can the calibration imatrix be

produced without llama.cpp, and does that change the result?**

What it is. An importance matrix (imatrix) records, per weight tensor, the

per-input-channel sum of squared activations over calibration text — it tells

llama-quantize where to spend bits. Normally you get it from llama-imatrix,

which needs the whole model resident in RAM+VRAM. Here it was instead computed

directly from the Hugging Face model in PyTorch, by registering forward hooks

on every matmul (attention, Mamba-2 in/out projections, router, shared experts,

and per-expert routed FFNs) and accumulating Σx² as calibration text streams

through. The generator is band-serialized: it loads a few decoder layers at a

time, runs all calibration chunks through them, caches activations, frees them,

and moves on — so peak VRAM is a few layers, not the whole model (this run

peaked at ~6.4 GB on a 16 GB card, vs the ~14 GB a full-resident forward needs).

That decouples imatrix generation from llama.cpp's memory model: you can calibrate

a model far larger than your GPU by streaming it in bands.

Everything else is identical. Same bf16 base, same APEX --tensor-type-file

recipe, same calibration_datav3, same 126×512 calibration windows. The only

difference from the standard i-quality file is which tool produced the imatrix.

The two quants therefore differ only in the tensors whose quantization

consumes an imatrix (the K-/I-quant expert, attention and Mamba weights); the

Q8_0 shared experts and F32 tensors are bit-identical.

Validation. The PyTorch imatrix matches llama-imatrix's output tensor-for-

tensor (368/368 names) with median per-tensor correlation 0.995. The handful

of lower-correlation tensors are the post-nonlinearity inputs (ffn_down_shexp,

attn_output) where transformers' naive Mamba scan and SiLU-gated intermediates

differ numerically from llama.cpp's kernels — a known wrinkle that does not affect

quality, since the quantizer only needs relative per-channel importance. The

proof is in the table above: the torch-imatrix quant scores PPL 8.864, inside

the ±0.11 error bar of both bf16 and the standard i-quality. The two imatrices are

interchangeable.

Pick whichever you like — they are equivalent in quality. The standard

i-quality is the reference; the torch build is provided for anyone who wants a

llama.cpp-free, VRAM-bounded path to the same result (e.g. calibrating very large

MoEs on modest hardware).

Attribution & licenses

See LICENSE (Apache-2.0) and NOTICE.

• Base: IBM (@ibm-granite) — granite-4.0-h-tiny (Apache-2.0)
• Engine: llama.cpp (@ggml-org) (MIT)
• APEX: Ettore Di Giacinto / LocalAI (@mudler) — localai-org/apex-quant (MIT)
• Calibration: Bartowski (@bartowski) — calibration_datav3

Unofficial community quantization; not affiliated with or endorsed by IBM.