---

license: apache-2.0

base_model: thinkingmachines/Inkling-Small

base_model_relation: quantized

library_name: gguf

pipeline_tag: image-text-to-text

tags:

• inkling
• gguf
• mixed-quantization
• imatrix
• multimodal
• dgx-spark

---

Inkling-Small — Mixed Quant GGUF

Two independently audited mixed-precision variants of

thinkingmachines/Inkling-Small, source revision

8cc5877b44d343f88b92086aa1fb72897950f06a.

Choose one variant directory to download. Each contains all of its shards,

checksums, tokenizer/processor/config assets, model card and validation reports.

> Multimodal serving (2026-09-10) and prefill optimization (2026-09-11, rounds 16–18): The MQ85GB variant serves text, PNG/JPEG image and 16 kHz WAV audio input with text output through ds4-dfm-rs on one DGX Spark: image requests pass on OpenAI Chat Completions, OpenAI Responses and Anthropic Messages, audio input on Chat Completions, and the separate eight-layer MTP-BF16 sidecar is checked on and off over HTTP. After three further prefill rounds (aligned IQ2_XXS/XS SoA and default chunk 1024), 8,192-token prefill reaches 328.16 tok/s and 2,048-token prefill 342.31 tok/s: medians of three fresh processes at the default 1024-token chunk, 64 greedy output tokens, MTP off. Versus the previously published chunk-512 default (312.88 / 327.86 tok/s) that is +4.9% / +4.4%. Each same-binary round and the cumulative compare checks 1,200,348 logits with zero differences and identical generated IDs. Runtime: ds4-dfm-rs@9c288a6, PR #30.

> Prefill optimization (2026-09-12, rounds 19–27): Nine further rounds on the same runtime raise 8,192-token prefill to 464.09 tok/s and 2,048-token prefill to 486.86 tok/s on the default path (medians of three fresh processes, chunk 1024, 64 greedy output tokens, MTP off; decode unchanged at 13.68 / 17.13 tok/s), +41.4% / +42.2% over the figures above and about 2× the campaign's starting 230.7 tok/s. Every retained round is byte-exact: each same-binary round and the cumulative compare check 1,200,348 logits with zero differences and identical generated IDs. Rounds 19–24 are merged (PR #31, PR #32); rounds 25–27 are in PR #33 (report). Round 27 adds an opt-in tensor-core prefill attention (DS4_INKLING_ATTN_HMMA=1) that reaches 501.87 tok/s at 8,192 tokens but is not byte-exact: it changes only fp32 summation order, yet this model amplifies any such change to relative RMS ≈0.1 on the logits and to different greedy continuations, so it stays off by default and the byte-exact prefill/decode parity above applies to the default path only.

MQ85GB runtime: ds4-dfm-rs integration and checks

cover serial CUDA serving on DGX Spark with text/image/audio input and text

output, plus the separate eight-layer MTP-BF16 sidecar. Short HTTP checks use

context 1,024; the guide records exact API/media limits and MTP on/off evidence.

The MQ85GB performance reports

(latest: rounds 25–27) record 8192- and

2048-token prefill and 64-token decode with MTP off, fresh-process samples and

exact full-logit/token parity against the retained ds4 control paths.

MQ89, llama.cpp compatibility, long-context

serving and MTP speedup remain unqualified by these checks.

| Variant directory | Bytes | Decimal GB | GiB | Shards |

|---|---:|---:|---:|---:|

| MQ89 | 95,446,595,200 | 95.446595 | 88.891569 | 6 |

| MQ85GB | 85,732,587,168 | 85.732587 | 79.844694 | 6 |

GB = 1,000,000,000 bytes; GiB = 1,073,741,824 bytes. MQ89 addresses the original

84–89 GiB weight budget. MQ85GB addresses the additional 84–86 GB budget,

saving 9.714008 GB for additional runtime headroom. MTP is separate and excluded.

Support my work

I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.

While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.

Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.

<a href="https://www.buymeacoffee.com/baekpica" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee" style="height: 60px !important;width: 217px !important;"></a> <a href="https://github.com/sponsors/Baekpica" target="_blank"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="Sponsor Baekpica on GitHub" style="height: 60px !important;width: 217px !important;"></a>

Download one variant

# MQ89
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ89/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ89 && sha256sum -c SHA256SUMS)

# MQ85GB
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ85GB/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ85GB && sha256sum -c SHA256SUMS)

Both retain all 888 main tensors, all 42 decoder layers, 256 experts in every

sparse layer, and complete native BF16 vision/audio ingress and attention.

general.architecture=inkling, inkling.tensor_layout=source-interleaved-v1,

GGUF quantization version 2. Gate/up interleaving and original tensor names/shapes

are unchanged. Read INKLING-CONTRACT.md before native work.

Precision differences

| Group | MQ89 | MQ85GB |

|---|---|---|

| Attention, vision/audio | Original BF16 | Original BF16 |

| L2 routed experts | BF16 | Q8_0 |

| Embedding/head, dense L0–1, shared experts | Q8_0 | Q8_0 |

| L3–39 routed gate/up | IQ2_XXS | IQ2_XXS |

| L3–39 routed down | Q2_K | IQ2_XS |

| L40 routed gate/up | Q4_K | Q3_K |

| L40 down and all L41 routed matrices | Q4_K | Q4_K |

| Router/norm/bias/scale/convolution/relative controls | Native | Native |

MQ85GB changes 40 precisions and preserves the other 848 tensor payloads

bit-exactly. Its L2 Q8 payloads also match the separately published Q8 reference.

All native payloads match original BF16/F32 source bytes. L2 Q8 is the deliberate

large capacity tradeoff; this variant does not retain that official NVFP4 BF16

exclusion. Detailed audits and results are in each variant directory.

Calibration and fidelity

Both use the same measured multimodal imatrix: 1,663 exact processed inputs /

1,578,255 valid decoder tokens. The balanced primary pass has importance weight

1.0; a real structured reasoning/tool coverage supplement has weight 0.1.

Effective text/code/image/audio shares are 45.215/21.433/23.817/9.535 percent.

The public calibration dataset

contains the exact inputs and complete source lineage/weighting details.

All L3–39 target experts were observed. L41 expert 123 remained unobserved;

its ordinary Q4_K no-imatrix payloads are identical in both variants. No missing

importance vector is fabricated. Shared calibration/ contains

NPZ/DAT importance and coverage. reproduction/ and

provenance/ record code, compiler/backend context and precise pins.

| Offline fidelity suite | MQ89 mean KL vs BF16 | MQ85GB mean KL vs BF16 | MQ89 top-1 agreement | MQ85GB top-1 agreement |

|---|---:|---:|---:|---:|

| short | 0.309541 | 0.382427 | 0.824 | 0.784 |

| extended | 0.133307 | 0.146076 | 0.918 | 0.920 |

These are small teacher-forced diagnostics (including real document/speech),

not task benchmarks or guarantees of equivalent quality. Individual probes,

direct variant comparisons and 32-token-budget generation checks are included.

Extended media were excluded from calibration; text rows were excluded from

the structured supplement, but overlap with historical primary text pools

cannot be fully ruled out. Offline GGUF weights were dequantized to BF16 on

B300 GPUs, so those timings/memory figures are not native quantized serving data.

The weight budget leaves room for caches and activations but does not prove

that a particular context fits on GB10. Seven global-attention layers alone

need about 28 GiB BF16 KV at 1,048,576 tokens, before local state, activations

and OS memory. Actual admission and serving measurements remain necessary.

Related artifacts

• Q8 reference and optional MTP-BF16.
• Exact calibration dataset.

The earlier MQ89 root-level paths remain available at historical revision

f091d3672dc69a1af77cad92b63cdeddc145a039; current main uses variant directories.

The MQ89 weight bytes and hashes did not change during this organization.

Source model is Apache-2.0; dataset sources retain their respective licenses.