---

license: apache-2.0

base_model: ibm-granite/granite-4.2-8b

library_name: gguf

tags:

• granite
• granite-4.2
• quantization
• imatrix
• llama.cpp
• benchmark

---

Granite 4.2 8B — G-Q5 mixed quantization + RTX 3060 optimization study


Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo

Two deliverables here:

1. G-Q5 — a custom mixed-precision GGUF of ibm-granite/granite-4.2-8b targeting near-bf16 fidelity at Q5-ish size/speed. Best-in-table accuracy in our testing.
2. A complete quantization/serving study for dense 8B models on consumer 12GB GPUs (RTX 3060): KL-divergence comparisons across all major community quants, KV cache dtype findings, max-context maps, speculative decoding results, and sampling pitfalls. All measured, nothing estimated.

G-Q5 quant recipe

| Tensor | Type |

|---|---|

| attn_q / attn_k / attn_output | Q5_K |

| attn_v | Q6_K |

| ffn_down | Q5_K |

| ffn_gate / ffn_up | Q4_K |

| token_embd | Q5_K |

| output | Q6_K |

• Quantized from bf16 (never requantize) with llama.cpp b062ba735.
• imatrix calibrated on ~600K tokens: wikitext-103 prose + local C++/TS/Python code (~80/20).
• Rationale: granite's untied embeddings on a 100K vocab are ~20% of params — the biggest accuracy lever. Attention projections carry the arch's unusual 1/128 attention scale.

Measured comparison (RTX 3060 12GB, CUDA, r=3 medians)

Fidelity = KL divergence vs bf16 over 30x512-token chunks of held-out wiki text.

| Quant | Size GiB | KLD ↓ | Same-top-p ↑ | tg128 | pp512 |

|---|---|---|---|---|---|

| Q4_K_M (official IBM) | 5.10 | 0.1796 | 86.0% | 55.6 | 1760 |

| Q4_K_M (bartowski) | 5.16 | 0.0996 | 90.4% | 53.5 | 1739 |

| IQ4_XS (bartowski) | 4.60 | 0.1149 | 87.9% | 59.9 | 1918 |

| Q4_K_L (bartowski) | 5.44 | 0.0911 | 90.4% | 53.0 | 1738 |

| G-Q5 (this repo) | 5.22 | 0.0809 | 91.2% | 50.5-53.8 | ~1700 |

Also tested and rejected:

• Q4KS-XL (Q4_K_S + Q5_K embd, imatrix): KLD 0.1333, tg 57-58 — strictly dominated by bartowski's IQ4_XS.
• EDGE (IQ2_XS middle-layer FFN, IQ3_XXS edge FFN, Q8 attention): 4.36G but slower than IQ4_XS (51.6 tg) with catastrophic KLD 0.622. IQ2-class kernels on Ampere MMQ are slow enough to eat the bandwidth savings, and same-top-p collapsed to 71%.

Key finding: official IBM quants are mediocre

The official Q4_K_M's KLD is nearly 2x worse than bartowski's free imatrix upgrade of the same quant. If you use granite from the official GGUF repo, you're leaving real accuracy on the table.

Serving findings for dense 8B on 12GB

Context limits (full offload, FA on)

| KV dtype | Max context |

|---|---|

| q4_0/q4_0 | 108K verified (109K+ OOM) |

| q8_0/q8_0 | <96K |

| f16/f16 | OOM past ~30K |

KV cost is architecture-determined: 40 full-attention layers x 8 KV heads x 128 head_dim = ~80KB/token at f16, ~22KB at q4_0. Hybrid-attention models (qwen3.5-style: SSM layers between attention) need 1/4 the KV per token — no weight quantization can close that gap.

KV dtype vs speed: no difference at practical depths

At 2.3K-token prompts: q4_0 KV = q8_0 KV within noise (pp ~1600, tg ~46 both). KV dtype only matters at deep fill (50K+). Use q8_0 KV for free quality; drop to q4_0 only to extend max context.

Agent-workload throughput: expect well below bench numbers

During a real agent session (hermes-core suite), median decode was 29 tok/s vs 50+ on clean llama-bench runs. Causes: growing context per task (attention cost scales with KV depth — long generations ran at 26 vs 32 for short ones) and per-request churn (127 requests in one suite). Plan capacity on the agent number, not the bench number.

Speculative decoding on granite

| Method | Result |

|---|---|

| ngram-mod self-speculative | +31% decode on repetitive/structured workloads, ~neutral on open-ended |

| granite-4.2-3b as classic draft | Net loss (-43%): drafter is also 40 layers, its pass costs more than it saves |

Same tokenizer != fast draft. Draft speedup requires draft model depth << target depth or a trained single-pass head (DFlash/EAGLE style).

Prefill notes

• Deep-fill prefill degrades gracefully: ~1635 tok/s at 89K tokens (vs 1750 cold).
• Decode at depth: 47 tok/s @ 89K fill, 13 tok/s @ 62K fill on 64K server — plan long-context serving accordingly.

Sampling: no significant difference found

We tested greedy (temp=0) vs IBM's official config (temp=1.0, top_p=0.95) across 10 varied prompts: 0/10 spirals for greedy vs 1/10 for official — no significant difference. An earlier single-run observation that temp 0 "causes reasoning loops" did not replicate; the loop we saw was on a heavily-quantized experimental variant, not this one. Use IBM's official config (what the model was tuned with), and give thinking mode a large budget (8K+) for math/code tasks.

Reproduce G-Q5

llama-quantize \
  --imatrix imatrix-granite42-mixed.gguf \
  --tensor-type-file tensor_types_G.txt \
  granite-4.2-8b-bf16.gguf granite-4.2-8b-G-Q5.gguf Q4_K_S 32

<details><summary>tensor_types_G.txt</summary>

attn_q=q5_k
attn_k=q5_k
attn_v=q6_k
attn_output=q5_k
ffn_down=q5_k
ffn_gate=q4_k
ffn_up=q4_k
token_embd.weight=q5_k
output.weight=q6_k

</details>

Files

• granite-4.2-8b-G-Q5-Q5_K_M.gguf — the quant (dominant type Q5_K; mixed per-tensor recipe above)
• imatrix-granite42-mixed.gguf — importance matrix used (reusable)
• calibration_mixed.txt — calibration corpus
• tensor_types_G.txt — per-tensor quantization map

HermesBench agent evaluation (G-Q5)

Ran the hermes-core tool-use suite (13 tasks, sequential, real Hermes agent + tool loop) on G-Q5.

Initial run: 6/13 (46%) — but 5 of the 7 failures were infrastructure, not the model. The llama-server crashed mid-run and 5 tasks burned their retries against a dead port. On rerun with a stable server:

| Task | Initial | Retry |

|---|---|---|

| file-and-terminal-self-serve | fail (conn error) | PASS |

| web-search-required | fail (conn error) | PASS |

| use-a-skill | fail (conn error) | PASS |

| memory-recall | fail (conn error) | fail — raw score 1.0 but failed verification gate |

| delegate-parallel-subtasks | fail (conn error) | PASS |

| todo-plan | fail (conn error) | PASS |

| browser-required | fail (61 tool calls, wrong outcome) | — |

Corrected score: 11/13 (85%). For reference from our July sweep: Qwen3.5-9B UD Q4_K_XL scored 77% (with MTP spec-decoding, turbo KV, 262K ctx) and Qwen3.6-35B-A3B UD IQ3_XXS scored 85%. Caveats before declaring granite the winner: the Qwen runs used a different serving regime, and this suite is small enough that one task flips the percentage by 8 points. Treat as "granite G-Q5 is in the same tier as the best local models we've tested on this GPU," not a strict ranking.

Tool-calling notes

• Granite uses a legacy XML tool-call format (<tool_call><function=name><parameter=...>); llama.cpp's --jinja parser handles it correctly — structured tool_calls come back properly.
• Parallel tool calls require "parallel_tool_calls": true in the request payload (llama.cpp default is off). Without it, only the first call in a turn survives.
• Single calls, no-tool restraint, and tool-result follow-up turns all work correctly out of the box.

Agent behavior notes

• Thinking-token heavy: several tasks ran 10-60 min due to verbose reasoning chains, not slowness of the weights.
• The memory-recall failure is instructive: granite solved the task correctly via a file shortcut but the grader required the memory tool path specifically. Rule-compliance, not capability.

Hardware context

All measurements: AMD Ryzen 9 5950X, RTX 3060 12GB, Fedora 43, llama.cpp b062ba735, CUDA. Localmaxxing runs approved: G-Q5, run ids cmt8zlca200ioli019ukksygh (G-Q5), cmt8zl9b300ikli01o2v42b0q (Q4KS-XL), cmt8wx9zq00ftli014hldgmc6 (official Q4_K_M baseline).