---

license: apache-2.0

base_model: ibm-granite/granite-4.1-30b

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- rocmfp4

- rocmfpx

- strix-halo

- gfx1151

- amd

- rocm

- granite

- quantized

---

Granite-4.1-30B — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of ibm-granite/granite-4.1-30b. Four quants, all built

from IBM's own BF16 GGUF (no re-conversion from safetensors), all verified on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory, ROCm 7.2.4 —

using the ROCmFPX llama.cpp fork, which adds AMD-native

FP4/FP8 tensor types that mainline llama.cpp does not have.

> ⚠️ These files require a ROCmFPX-capable llama.cpp build. They will not load in stock

> llama.cpp / Ollama / LM Studio — the Q4_0_ROCMFP4_ and Q_0_ROCMFPX* tensor types are not

> in mainline.

Variants — all four in this repo

| file | ftype | size | BPW | token_embd | decode | pp512 |

|---|---|---|---|---|---|---|

| granite-4.1-30b-Q4_0_ROCMFP4_COHERENT.gguf | 102 | 15.23 GiB | 4.53 | Q6_K | 13.07 t/s | 292.81 t/s |

| granite-4.1-30b-Q6_0_ROCMFPX_AGENT.gguf | 114 | 24.98 GiB | 7.43 | Q8_0 | 7.73 t/s | 210.63 t/s |

| granite-4.1-30b-Q8_0_ROCMFPX.gguf | 111 | 27.74 GiB | 8.25 | Q8_0 | 7.05 t/s | 297.23 t/s |

| granite-4.1-30b-Q8_0_ROCMFPX_AGENT.gguf | 115 | 28.19 GiB | 8.39 | Q8_0 | 6.87 t/s | 309.14 t/s |

Which to pick: the 4-bit (102) unless you specifically need 8-bit fidelity — it is **1.85×

faster** and less than half the size. The AGENT variants keep more attention-output tensors at

higher precision, which helps tool-calling and (on models that have one) speculative-draft

acceptance; Granite-4.1-30b ships no draft head, so here AGENT costs a little speed for a

little precision.

⚠️ Q6_0_ROCMFPX_AGENT is not a middle rung — it lands at 7.43 BPW and performs like the

8-bits. Choose it for precision, not for size.

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified, ROCm 7.2.4.

llama-bench -n 300 -p 512 -r 3, warm-up discarded, idle box. Median of 3; all three runs shown.

| quant | run 1 | run 2 | run 3 | median |

|---|---|---|---|---|

| 102 | 13.0720 | 13.0714 | 13.0671 | 13.07 |

| 114 | 7.72874 | 7.72838 | 7.72677 | 7.73 |

| 111 | 7.03671 | 7.05450 | 7.05481 | 7.05 |

| 115 | 6.90177 | 6.86858 | 6.87124 | 6.87 |

These are the deployed numbers. Granite-4.1-30b ships no MTP/EAGLE/draft head, so there is no

speculative-decoding multiplier being left on the table.

Why you can trust the speed figures

Weights stream once per token, so t/s × file_GB gives effective memory bandwidth. Against Strix

Halo's ~256 GB/s peak:

| quant | GB/s | % of peak |

|---|---|---|

| 102 | 213.7 | 83.5% |

| 114 | 207.3 | 81.0% |

| 111 | 210.0 | 82.0% |

| 115 | 208.0 | 81.2% |

All four sit at 81–83% of peak, and the 4-bit/8-bit speed ratio (1.85×) tracks the inverse size

ratio (1.82×). That is the signature of a purely bandwidth-bound dense model running at the

hardware ceiling — nothing is misconfigured or spilling to CPU.

Verification

Every artifact was checked individually:

• Loaded with -ngl 999 -c 4096 -fa on -fit off
• Correctness 3/3 at max_tokens: 1024 — 17*23 → 391, capital of Japan → Tokyo,

days in 2024 → 366

• Size vs --dry-run projection — each file exceeds its projection by a constant GGUF-header

delta (+3,576,564 B; +3,579,186 B for the 4-bit), which is the truncation check

• Head protection audited by exact tensor name, not by trusting the flag (see below)

Exact byte sizes:

16348341472  granite-4.1-30b-Q4_0_ROCMFP4_COHERENT.gguf
26820404448  granite-4.1-30b-Q6_0_ROCMFPX_AGENT.gguf
29785777376  granite-4.1-30b-Q8_0_ROCMFPX.gguf
30272316640  granite-4.1-30b-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

granite-4.1-30b has tie_word_embeddings = true — there is **no output.weight tensor at

all. That means --output-tensor-type is a silent no-op** on this model, and

--token-embedding-type is the only flag that actually protects the head:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  granite-4.1-30b-bf16-00001-of-00005.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Verify it landed by reading the finished file's token_embd.weight type — do not assume it worked

because the flag was accepted. Also: do not grep output.weight when auditing, because it

false-matches blk.N.attn_output.weight and will make a healthy build look damaged.

Model

Dense, 64 layers, hidden 4096, FFN 32768, GQA 32/8 heads, vocab 100352, context 131072.

Source: ibm-granite/granite-4.1-30b-GGUF BF16 shards (00001-of-00005), quantized directly —

no safetensors re-conversion.

Credits

• Base model: IBM Granite (Apache 2.0)
• Quantization stack: ROCmFPX