---

license: apache-2.0

base_model: thinkingmachines/Inkling-Small

language:

- de

- en

tags:

- gguf

- llama.cpp

- moe

- mixture-of-experts

- expert-pruning

- reap

- german

pipeline_tag: text-generation

---

Inkling-Small-REAP-137B-A12B-de (GGUF)

A REAP expert-pruned variant of thinkingmachines/Inkling-Small

(276B-A12B → 137B total / 12B active), provided as GGUF quantizations.

Variants in this repo

| Variant | Size | BPW | Notes |

|---|---|---|---|

| IQ4_XS (imatrix) | 72.0 GB | ~4.3 | Recommended — importance-matrix-weighted, smaller and typically better than plain Q4_K_M |

| Q4_K_M | 81.5 GB | 4.84 | Plain k-quant, no imatrix |

| imatrix.dat | 217 MB | — | Importance matrix (538 entries) — build your own quants with llama-quantize --imatrix |

The importance matrix was computed on a German-heavy DE/EN mix matching the

pruning calibration (German multi-register human text, German model outputs,

English code and math), so the quantization error budget follows the same

language priorities as the expert pruning. Both variants are split into ≤45 GB

parts; keep the parts of one variant in the same directory and load the first

part.

The idea: produce a sparse model in the 100–140B class with a

moderate quantization level (Q4_K_M, no aggressive sub-4-bit tricks), so that

the quality loss budget is spent once on expert pruning and once on a

well-understood quantization — not twice on extreme compression. We plan to

spot-benchmark it against dense/sparse models of similar deployed size

(e.g. Qwen3.5-122B-class models).

⚠️ Language disclaimer: the expert-pruning calibration was **optimized for

German and English** (German-heavy multi-register calibration mix). Losses in

other languages are presumably significantly higher — experts that were

mostly used by other languages were preferentially removed. If you need

multilingual coverage beyond DE/EN, use the original Inkling-Small.

What was done

• Method: REAP (Router-weighted Expert Activation Pruning,

Lasby et al., ICLR 2026) — one-shot,

no retraining. Per MoE layer, the 128 of 256 routed experts with the lowest

router-weighted activation saliency were removed; router rows / e-score bias

sliced accordingly. Shared experts, attention, dense layers, embeddings and

MTP weights are untouched.

• Calibration: ~4M tokens, 56 % German (human-written text across 10

registers — news, legal, parliamentary, scientific, encyclopedic, spoken,

easy language, essays, social, narrative — plus German instruction/chat data

and German outputs of frontier models), 44 % English code / reasoning /

tool-calling / chat.

• Quantization: llama.cpp Q4_K_M from BF16.

Calibration only collects routing statistics; no calibration text enters the

weights.

Internal retention probes (not benchmarks)

Teacher-forced streaming probes against the BF16 base model (ΔNLL in nats,

top-1 agreement with base). Spot checks, not standardized benchmarks:

| Probe | ΔNLL vs. base | Top-1 agreement |

|---|---|---|

| German prose | +0.39 | 66.7 % |

| German factual text | −0.18 | 77.2 % |

| English prose | +0.72 | 73.0 % |

| Python | +0.68 | 81.0 % |

For comparison: an English-only-calibrated pruning of the same ratio degraded

German prose by +1.67 nats. Proper benchmark results (vs. Qwen3.5-122B-class

models) are planned.

Usage

Requires a llama.cpp build with Inkling support — currently the

add-inkling PR #25731

until it is merged:

git clone -b add-inkling https://github.com/danielhanchen/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/llama-cli -m Inkling-Small-REAP-137B-A12B-de-Q4_K_M-00001-of-00002.gguf \
  -p "Erkläre den Unterschied zwischen Bundesrat und Bundestag." -n 300

The split GGUF parts must sit in the same directory; llama.cpp loads them

automatically from the first part.

Limitations

• DE/EN optimized — other languages likely degrade well beyond typical

REAP losses (see disclaimer above).

• Code/tool-calling gives up a little quality relative to an English/code-only

calibration (that trade-off was chosen deliberately).

• Text-only: vision/audio towers of the multimodal original are not part

of this GGUF.

• No MTP speculative decoding in this GGUF (llama.cpp Inkling MTP support

pending). The underlying pruned safetensors checkpoint retains the full MTP

stack.

• Pruned models remain probabilistic compressions of the original — verify

before production use.

About the base model: Inkling-Small

> ⚠️ Carry-over disclaimer: Everything in this section describes the

> original thinkingmachines/Inkling-Small

> (276B-A12B, BF16, multimodal). This repo contains a **50 %-expert-pruned,

> Q4_K_M-quantized, text-only** derivative — the properties and scores below

> do not all carry over. Treat every number as an upper bound, not a

> claim about this model. Multimodal inputs are absent here entirely, and

> multilingual ability outside German/English was deliberately traded away

> during pruning calibration.

Key facts (original)

| Property | Original Inkling-Small |

|---|---|

| Model type | Multimodal autoregressive transformer (text/image/audio → text) |

| Architecture | 42-layer decoder, sparse MoE FFN: top-6 of 256 routed experts + 2 shared; hybrid local/global attention |

| Parameters | 276B total / 12B active (this repo: 137B total / 12B active) |

| Context length | up to 1M tokens (long-context behavior of the pruned model is untested) |

| Numerics | BF16 and NVFP4 (this repo: Q4_K_M GGUF) |

| Languages | English, with general multilingual capabilities (this repo: DE/EN-calibrated) |

| Training data | Public, third-party-acquired and synthetic text/image/audio/video (see original card) |

| MTP | 8 speculative multi-token-prediction layers (preserved in our safetensors, not in this GGUF) |

| License | Apache-2.0 |

Selected original benchmark scores

Scores of the unpruned BF16 original, from the official model card (where

it is compared against Qwen3.5-397B-A17B, MiMo V2.5, Minimax M2.7 and closed

models — see there for the full 10-model table). **None of these numbers have

been re-measured for this pruned+quantized variant.**

| Benchmark | Original Inkling-Small |

|---|---|

| SWEBench Verified | 80.2 % |

| Terminal Bench 2.1 (best harness) | 64.7 % |

| SciCode | 48.7 % |

| GPQA Diamond | 89.5 % |

| AIME 2026 | 95.5 % |

| HLE (text only / with tools) | 31.6 % / 47.8 % |

| ARC-AGI-1 / ARC-AGI-2 | 84.0 % / 40.1 % |

| BrowseComp (with context mgmt) | 77.4 % |

| IFBench | 82.2 % |

| Global-MMLU-Lite | 86.7 % (expect larger drops off DE/EN here) |

| SimpleQA Verified | 20.6 % |

| StrongREJECT (safety) | 98.4 % |

Multimodal scores (MMMU Pro, Charxiv, MMAU, VoiceBench …) are omitted: the

vision/audio towers are not part of this GGUF.

Deployment of the original

The original model has day-0 recipes for SGLang, vLLM, TokenSpeed, Unsloth and

Hugging Face transformers (links in the original card), including MTP

speculative decoding. This GGUF instead targets llama.cpp (see Usage

above); our pruned safetensors checkpoint (TML layout, MTP intact,

vLLM/SGLang-compatible) is not published here — open a discussion if you need

it.

Safety, bias and limitations (inherited)

The original card documents safety training, known biases, risks and legal

terms — all of that applies here at best unchanged and plausibly degraded:

expert pruning is a lossy intervention whose effect on safety behavior and

refusal calibration has not been separately evaluated. Review the original

card's Safety / Bias / Legal sections before production use.

Provenance

• Base: thinkingmachines/Inkling-Small (Apache-2.0)
• Pruning: REAP streaming pipeline

(calibration ran block-wise on a single RTX 5090 workstation without ever

loading the 532 GB model into memory)

• Per-layer pruned expert lists: see reap_pruning.json in this repo

@inproceedings{lasby2026reap,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author={Lasby, Lazarevich, Sinnadurai, Lie, Ioannou, Thangarasa},
  booktitle={ICLR}, year={2026}
}

---

Deutsch (Kurzfassung)

REAP-geprunte Variante von Inkling-Small: pro Layer wurden die 128 von 256

Experten mit der geringsten router-gewichteten Aktivierungs-Salienz entfernt

(one-shot, kein Training), anschließend Q4_K_M-Quantisierung. Die Kalibrierung

war bewusst deutschlastig (56 %, menschliche Texte über zehn Register plus

deutsche Instruktionsdaten): deutsche Prosa verliert nur +0,39 nats gegenüber

dem Original (statt +1,67 bei englischer Kalibrierung), deutscher Sachtext

liegt auf Originalniveau. **In anderen Sprachen als Deutsch und Englisch sind

die Verluste mutmaßlich deutlich höher.** Geplant sind Stichproben-Benchmarks

gegen Modelle der Qwen3.5-122B-Klasse.