---

license: apache-2.0

base_model: internlm/Intern-Decision-4B

base_model_relation: quantized

library_name: gguf

pipeline_tag: image-text-to-text

tags:

• gguf
• llama.cpp
• decision-making
• structured-prediction
• quantized

---

Intern-Decision-4B GGUF (F16 / Q8_0 / Q5_K_M / Q4_K_M)

GGUF conversions of internlm/Intern-Decision-4B

with a decision-level fidelity evaluation on the official Intern-Decision benchmarks: every

quantization is compared, decision by decision, against the original BF16 checkpoint run in the

same environment.

This is an unofficial community conversion; it is not affiliated with InternLM / Shanghai AI Lab.

Recommended file: Intern-Decision-4B-Q5_K_M.gguf — 3.07 GB, ~3.9 GB VRAM at 8K context,

98.5% of decisions identical to BF16, unchanged average accuracy.

Files

| File | Size | SHA-256 |

|---|---:|---|

| Intern-Decision-4B-F16.gguf | 8.42 GB | 97525f7157cda0b44b73b714ee2fa7297c94e3cb56b5981fd80de365f496bcca |

| Intern-Decision-4B-Q8_0.gguf | 4.48 GB | 86198e50f22d6a43202b3e73fdb74a0bd7caece1136975ac3ecf530787a50813 |

| Intern-Decision-4B-Q5_K_M.gguf | 3.07 GB | a322e305daf5f78c3865d9dd812a48ccd276035d3882af6f09f5b1e6e002b359 |

| Intern-Decision-4B-Q4_K_M.gguf | 2.71 GB | 3c3be9bd66008743fc24a763f28e2de51fb2417d2e4406ae59a3015ec7eaeaab |

| mmproj-Intern-Decision-4B-F16.gguf | 0.67 GB | beaeba2ae864a437ab4ef681a9e58b5ab7e1882e23eb3c66f0d30911178715a9 |

| decision_gguf.py | | Reference scorer (see below) |

| eval/ | | Evaluation runner, comparison script and summary results |

The vision projector is provided for completeness; all evaluations below are text-only.

This is a scoring model, not a chat model

Intern-Decision does not generate answers. A request (state + named questions with options) is

compiled into a chat transcript whose assistant turn is a JSON skeleton with one <decision>

placeholder per field; the model's logits at the position immediately before each placeholder

are restricted to that field's option symbols (A, B, …), softmaxed, and calibrated with

temperature scaling (T = 1.99241824, fitted upstream on BF16 and left unchanged here).

Using these files through a chat UI or llama-server completion endpoint will not reproduce the

model's behaviour. Use decision_gguf.py, which takes the prompt compiler, chat template, symbol

mapping and calibration directly from the official inference.py and only runs the forward pass on

llama.cpp (no PyTorch needed):

pip install llama-cpp-python huggingface_hub transformers numpy
python decision_gguf.py --gguf Intern-Decision-4B-Q5_K_M.gguf --request request.json

request.json uses the upstream format, e.g.

{"state": "The customer was charged twice and asks for the extra payment back.",
 "questions": {"team": {"type": "choice", "instructions": "Which team should handle this request?",
                        "criteria": {"billing": "Payments and refunds", "delivery": "Shipping and delivery"}},
               "refund_requested": {"type": "noul", "instructions": "Is the customer asking for a refund?"}}}

Each <decision> position is scored with one batched forward over its full prefix. Feeding the

prompt incrementally (continuing the KV/recurrent state between decision positions) is

mathematically equivalent but, for K-quants, changed logits by up to ~0.6 because llama.cpp picks

different quantized kernels for small batches — enough to flip near-tie decisions.

Conversion

• Source: internlm/Intern-Decision-4B revision 0e5e6aa7d6d750e2b1504ba11a8136cb58aeb3cd.

All 14 checkpoint files match the SHA-256 hashes recorded for the 4B model in the official

benchmarks/temperature-presets.json, i.e. this is the same checkpoint the published results used.

• llama.cpp d81235049384534c167caea52b85a694f6103d14 (CPU build), architecture qwen35.
• python convert_hf_to_gguf.py <hf_dir> --no-nextn --outtype f16, then llama-quantize to each type

(no imatrix). --no-nextn is required: config.json declares mtp_num_hidden_layers: 1 but the

checkpoint contains no MTP weights, and without the flag the GGUF declares a block with no tensors

and fails to load.

• mmproj: convert_hf_to_gguf.py <hf_dir> --mmproj --outtype f16.
• Tokenizer: <decision> is preserved as a single special token (id 248077); in every spot check (benchmark

self-checks and test requests) the GGUF tokenizer reproduced the HF token ids exactly.

Evaluation

Official bundle from InternLM/Intern-Decision at

commit 3572c8a68b5df5dafe02d0e093989ba8ec0183bc: the seven accuracy-v1 suites (10,751 rows /

12,351 decisions) and the 96-case known-distribution calibration pilot, scored with the

repository's own src.eval.jev.evaluate, upstream Jevbench scoring, run_pilot.predict_inputs and

score_known_distribution.score_benchmark.

• Baseline: original BF16 weights through the official HF backend (--backend hf,

transformers 5.14.1, SDPA).

• GGUF: a backend subclassing the official HFBackend (identical encode()), forward pass via

llama-cpp-python 0.3.36 with CUDA, all layers on one GPU.

• Hardware: NVIDIA Tesla T4 16 GB (Kaggle), one model per GPU.

Accuracy (%)

| Suite | n | BF16 (HF) | F16 | Q8_0 | Q5_K_M | Q4_K_M |

|---|---:|---:|---:|---:|---:|---:|

| Jevbench-Easy | 48 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |

| Jevbench-Original | 72 | 98.61 | 98.61 | 98.61 | 100.00 | 100.00 |

| Jevbench-Hard | 111 | 72.97 | 72.97 | 72.97 | 72.07 | 70.27 |

| Typed Decision | 2,000 | 80.50 | 80.75 | 80.50 | 81.35 | 79.80 |

| ToolACE | 310 | 96.45 | 96.45 | 96.45 | 96.45 | 96.45 |

| AG News | 7,600 | 90.80 | 90.82 | 90.80 | 90.68 | 90.39 |

| WildJailBreak | 2,210 | 89.82 | 89.95 | 89.86 | 89.14 | 89.00 |

| Average (7 suites, unweighted) | | 89.88 | 89.94 | 89.88 | 89.96 | 89.42 |

For reference, the upstream model card reports 90.02 for Intern-Decision-4B using the XTuner

backend; upstream documents HF-backend results as a different execution setting.

Decision-level agreement with BF16 (12,351 decisions)

| | Same argmax as BF16 | BF16 right → quant wrong | BF16 wrong → quant right | wrong → different wrong | Mean / p99 \|Δ top-prob\| |

|---|---:|---:|---:|---:|---:|

| F16 | 99.69% | 13 | 22 | 3 | 0.003 / 0.027 |

| Q8_0 | 99.62% | 22 | 23 | 2 | 0.004 / 0.045 |

| Q5_K_M | 98.51% | 91 | 84 | 9 | 0.014 / 0.182 |

| Q4_K_M | 97.13% | 201 | 136 | 18 | 0.026 / 0.297 |

F16 involves no quantization, so its 0.31% disagreement is the floor set by the backend change

(llama.cpp vs PyTorch kernels). Q8_0 is indistinguishable from that floor. Q5_K_M changes ~1.2%

more decisions, with gains and losses nearly balanced. Q4_K_M loses net accuracy, most visibly on

Jevbench-Hard and AG News.

Calibration (fixed T = 1.99241824)

| | Jevbench-Hard Brier ↓ | Jevbench-Hard ECE ↓ | Pilot TVD (raw → calibrated) ↓ | Pilot excess Brier (calibrated) ↓ |

|---|---:|---:|---:|---:|

| BF16 (HF) | 0.347 | 0.080 | 0.269 → 0.216 | 0.105 |

| F16 | 0.349 | 0.074 | 0.268 → 0.214 | 0.104 |

| Q8_0 | 0.350 | 0.073 | 0.271 → 0.216 | 0.104 |

| Q5_K_M | 0.347 | 0.075 | 0.278 → 0.218 | 0.109 |

| Q4_K_M | 0.372 | 0.065 | 0.257 → 0.209 | 0.096 |

Jevbench-Hard has 111 items, so its Brier/ECE are noisy. Q8_0 and Q5_K_M stay close to BF16 on

all calibration metrics with the upstream temperature unchanged.

Memory (single T4, n_ctx = 8,256, all layers offloaded)

| | Load | Peak during suites |

|---|---:|---:|

| BF16 (HF, SDPA) | 8.9 GB | 12.4 GB |

| F16 | 9.0 GB | 9.3 GB |

| Q8_0 | 5.1 GB | 5.4 GB |

| Q5_K_M | 3.9 GB | 4.2 GB |

| Q4_K_M | 3.4 GB | 3.7 GB |

Smaller contexts need less. Latencies measured on the T4 are not reported as representative

(BF16 has no native support on Turing and the HF path used reference kernels for the linear-attention layers).

Reproducing

eval/worker.py (runs inside a checkout of the official repository) and eval/analyze.py

reproduce the tables; eval/comparison.json holds the summary numbers and sample flips.

License

Apache-2.0, inherited from Intern-Decision-4B, which is derived from Qwen3.5; LICENSE and

LICENSE-QWEN are retained. The weights are unmodified apart from format conversion and

quantization.