---

license: apache-2.0

base_model:

• OrionLLM/GRM-2.6-Plus-0628

base_model_relation: quantized

pipeline_tag: image-text-to-text

library_name: gguf

tags:

• gguf
• imatrix
• reasoning
• quantized
• qwen3_5
• mtp
• llama.cpp
• agentic

---

GRM-2.6-Plus-0628 — Reasoning-imatrix GGUF

Quantized GGUF builds of OrionLLM/GRM-2.6-Plus-0628 (a 27B agentic reasoning model, Qwen3.5/3.6 architecture with an MTP layer), calibrated with a reasoning-domain importance matrix rather than a generic one.

The whole point of these builds: the importance matrix was computed on a reasoning corpus (chain-of-thought, expository QA, math word problems), so the low-bit quants preserve the model's step-by-step reasoning and agentic behaviour where it matters most. In a single-variable A/B against a generic imatrix, this reasoning imatrix wins on both reasoning and general text (see below).

Every quant keeps the MTP / nextn layer (blk.64) in Q4 or above for speculative decoding.

Quant table

Perplexity measured on held-out data with identical parameters (-c 512 -ngl 99 --flash-attn auto). reasoning = held-out reasoning corpus (zero overlap with the imatrix calibration set); wikitext = wikitext-2 test. The Δ vs Q6 columns give each quant's perplexity gap above the Q6_K ceiling (near-lossless ≈ bf16), in percent — the honest "how far from lossless am I" number.

| Quant | BPW | Size | PPL reasoning | PPL wikitext | Δ vs Q6 (reas.) | Δ vs Q6 (wiki.) |

|---|---|---|---|---|---|---|

| IQ2_M | 3.18 | 10.1 GB | 6.0669 | 7.4683 | +6.61 % | +11.98 % |

| IQ3_XXS | 3.50 | 11 GB | 5.8443 | 7.0726 | +2.70 % | +6.05 % |

| Q3_K_M | 4.15 | 13 GB | 5.7977 | 6.8225 | +1.88 % | +2.30 % |

| IQ4_XS | 4.74 | 15.1 GB | 5.7200 | 6.7104 | +0.51 % | +0.62 % |

| Q4_K_M | 5.16 | 16.4 GB | 5.7098 | 6.7081 | +0.34 % | +0.58 % |

| Q4_K_XL | 5.24 | 17.9 GB | 5.7067 | 6.7120 | +0.28 % | +0.64 % |

| Q5_K_M | 5.72 | 19 GB | 5.6939 | 6.6672 | +0.06 % | −0.03 % |

| Q6_K | 6.56 | 21 GB | 5.6907 | 6.6691 | — | — |

> Q4_K_XL = Unsloth-style per-tensor recipe + imatrix, from community feedback. It replays Unsloth's Dynamic (UD-Q4_K_XL) per-tensor bit mixture on GRM, rebuilt on this reasoning imatrix — protecting the hybrid path (SSM output projections at Q8_0, gates at F32, attention/ffn_down at Q6_K) while the bulk stays Q4_K. The MTP layer (blk.64) is capped uniformly at Q4_K — the model's speculative-decoding acceptance saturates by Q4, so higher-precision MTP bits buy nothing — a ~27 MB trim with no effect on output quality (perplexity and generation never route through blk.64).

> Deployable tier (IQ2_M → Q4_K_M) — DeltaNet output protection. Every quant from IQ2_M up to Q4_K_M keeps the recurrent output projection (ssm_out) at Q8_0 while the bulk stays at each tier's base precision. GRM's hybrid path is gated DeltaNet, not a classic SSM: ssm_out has no upstream norm and no state accumulation, which makes it topologically the most quantization-sensitive tensor in the network — the damage concentrates in worst-case per-token divergence (Max-KLD), not average perplexity. Protecting it costs only +0.15–0.26 bpw, localized to one tensor. The mean-PPL payoff follows a clean gradient: real-ish at the bottom (IQ2_M trims its Δ-vs-Q6 from +7.40 % → +6.61 % reasoning, +13.28 % → +11.98 % wikitext), fading into the ±0.7 % noise floor by IQ4_XS/Q4_K_M — but the point is uniform worst-case insurance on the recurrent path across the whole deployable range, and it makes the published IQ4_XS bit-identical to the 24 GB reference deployment (see "Which one to pick"). Same principle as the ssm_out Q8_0 already inside Q4_K_XL; Q5_K_M/Q6_K sit at the bf16 ceiling everywhere and need no special protection.

PPL is monotonic in bit-width on both corpora, bar a single sub-noise-floor wikitext blip (Q4_K_M edges Q4_K_XL by 0.004, ~10× under the ±0.043 error bar — pure jitter; on reasoning the order holds and Q4_K_XL stays ahead). The imatrix behaves.

Which one to pick — by GPU

• 16 GB (RTX 4080 / 5070Ti) → IQ3_XXS, text-only. At 128K context (KV Q4_0), 512 batch size, the 11.1 GB DN build plus cache fits inside the 16 GB budget — but headroom is slim with a desktop running (the DN protection adds ~0.5 GB over the pre-DN IQ3_XXS), so keep context at or below 128K on a 16 GB card.
• 24 GB (RTX 3090 / 4090) → IQ4_XS. This is the build these quants were tuned around, and the published file is the DeltaNet-protected build (ssm_out Q8_0, 4.74 bpw). At 15.1 GB it still runs *262K context (KV cache Q4_0) plus the Q8 vision mmproj* on a single 24 GB card — measured at ~23.9 / 24 GiB used (≈0.66 GiB headroom), real config, no offload. You get Q4-grade quality (inside the noise floor on reasoning), full agentic + vision, and a quarter-million-token window on one 3090. That combination is the whole reason this quant exists.
• 32 GB (RTX 5090) → Q5_K_M. With the VRAM to spare, take the near-lossless tier. Q5_K_M is statistically identical to Q6_K (see plateau above) at 2 GB less — there is nothing visible to gain from Q6, so Q5 is the ceiling in practice.
• Q4_K_M — balanced default if you'd rather have the plain K-quant than IQ4_XS; comfortable on 24 GB at large (non-extreme) context.
• IQ3_XXS — the value pick: reasons and tool-calls reliably in 11 GB, fits a lot of context on smaller cards. Accepts a real but modest ~2.7 %/6 % PPL cost.
• IQ2_M — minimal footprint (10.1 GB) for tight-VRAM setups; still a functional agent, at the largest quality cost in the lineup.
• Q6_K — the absolute ceiling if you want it on paper; buys nothing measurable over Q5_K_M.

imatrix methodology

• Calibration corpus: reasoning domain (chain-of-thought / expository QA / math word problems), block-shuffled so a sequential read is representative of the whole set.
• Coverage: 400 chunks × 512 tokens.
• Source model for statistics: Q6_K (full-GPU, near-lossless) — the imatrix is a coarse per-channel activation statistic, so a Q6_K source is equivalent to bf16 but computes in minutes.
• MTP handling: the blk.64 nextn layer is never exercised in a normal forward pass, so it collects zero imatrix entries. Very-low-bit IQ quants require a per-tensor imatrix and would bail on blk.64; it is forced to Q4_K (a K-quant that needs no imatrix), keeping the MTP layer intact while the rest of the network uses IQ + reasoning imatrix.
• The imatrix file is included (grm-2.6-plus-0628-reasoning.imatrix, GGUF format) — download it to build your own quants (custom recipes, dynamic-XL, etc.) on this reasoning calibration instead of a generic one. Remember the blk.64.=q4_K override above for very-low-bit IQ builds.

Why a reasoning-domain imatrix

The importance matrix is a coarse per-channel activation statistic: what matters is which activations it samples, not raw token count. Calibrating on a reasoning corpus concentrates the statistic on the channels this model actually uses for step-by-step work. In controlled testing this held up without any cost on general text — wikitext perplexity was on par with a generic calibration — so the reasoning focus is a targeted choice, not a trade-off.

Usage (llama.cpp)

The correct chat template (Qwen3.6 XML tool-calling, <function=...>) is embedded in every GGUF — just pass --jinja, no --chat-template-file needed.

The example below is the real 24 GB config — IQ4_XS at 262K context with vision.

llama-server \
  --model grm-2.6-plus-0628-IQ4_XS-reasoning-imat.gguf \
  --mmproj Qwen3.6-27B-mmproj-hybrid-Q8_0-F16.gguf \
  --host 127.0.0.1 \
  --port 5000 \
  --ctx-size 262144 \
  --n-gpu-layers 65 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --batch-size 512 \
  --ubatch-size 512 \
  --rope-freq-base 10000000 \
  --parallel 1 \
  --slots \
  --jinja \
  --flash-attn auto \
  --image-min-tokens 1024 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1

The --spec-type draft-mtp flags enable speculative decoding via the built-in MTP layer (same GGUF, no separate draft model). For the 16 GB tier, swap in grm-2.6-plus-0628-IQ3_XXS-reasoning-imat.gguf, --ctx-size 131072, and drop --mmproj.

Vision

These are vision-language weights. To enable image input, pass a Qwen3.6-27B multimodal projector with --mmproj. A matching one is here:

DAXZEIT/Qwen3.6-27B-mmproj-hybrid-Q8_0-F16-gguf

It's a single hybrid file: most tensors are Q8_0, but ffn_down is kept at F16 because a full-Q8 down-weight doesn't hold up — hence the hybrid-Q8_0-F16 name (in practice people just call it the Q8 mmproj). ~601 MB on disk, ~629 MB of VRAM in the 24 GB / 262K config above.

Credits

• Base model: OrionLLM/GRM-2.6-Plus-0628 (Apache-2.0)
• Quantization: llama.cpp
• Q4_K_XL per-tensor recipe: adapted from Unsloth's Dynamic (UD) quant mixture
• Reasoning imatrix + builds: DAXZEIT