---

license: apache-2.0

base_model:

- thinkingmachines/Inkling-Small

library_name: llama.cpp

tags:

- gguf

- layer-library

- multi-stage

- ik_llama.cpp

- dreamcatcher

- xautonomics

- inkling

---

Inkling-Small-UD-Q4_K_M.LAYR.GGUF

A per-layer GGUF library of Unsloth's unsloth/Inkling-Small-GGUF (UD-Q4_K_M). These are the same quantized weights, sliced once into one file per transformer block plus the shared parts, so any contiguous layer window A, B) can be assembled at load time. It is built for [dreamcatcher, xAutonomics' fork of ik_llama.cpp, which serves a library either in a single process or split across hosts.

| | |

|---|---|

| base model | thinkingmachines/Inkling-Small (Thinking Machines Lab) · mixture-of-experts, 276B total / 12B active parameters |

| source quant | unsloth/Inkling-Small-GGUF · UD-Q4_K_M · pinned revision 1a19ef82 · 5 shards, 151.4 GiB |

| architecture | inkling · 42 blocks (2 leading dense) · 256 routed experts, 6 active + 2 shared · sliding window 512 · no NextN/MTP block |

| files | 45 (embd, layer, other, output) · 151.9 GiB |

| integrity | manifest.json: blake2b-128 per whole file (hash_scope: whole-file) and per tensor |

| sliced with | layer_distribution.slice at dreamcatcher commit 11c4f7a5 (also in SLICER_COMMIT); nothing was re-quantized |

Support status

This is the text-only Inkling port tracked in dreamcatcher issue #43. What we have and have not validated, as of publication:

Validated

• Cross-lineage envelope on the real 152 GB model (the monolithic UD-Q4_K_M source of this library, measured with KL-divergence against a banded oracle produced by the upstream llama.cpp draft build, ggml-org/llama.cpp#25731): the D2 masked attention path reaches 83.02 % top-1 agreement and the D3 banded attention path 83.87 %, both inside the pre-registered 17.08 % working band (the oracle build's own masked-vs-banded self-disagreement).
• Remote routed experts: the expert-server path is byte-exact against in-process experts.
• Stage ring: a head/tail llama-stage-runner ring runs token-identical to the monolith on the synthetic Inkling fixture that carries this model's scalar hyperparameters.

Not yet validated / in progress

• Greedy token parity against the upstream llama.cpp draft (#25731) build on this model — tracked in #43.
• This library has been hash-verified against its source tensors (every file and every tensor, see manifest.json), but the numbers above were measured on the monolithic source GGUF; the library-assembled load of this model is not yet on record.

Unvalidated

• Multimodal (audio/vision): the port is text-only. The mmproj-* projector files of the source repository are not included and are not supported.
• The measurements above were taken with the fleet's CUDA build; other backends (Vulkan, SYCL, CPU-only) are not measured for this architecture.

Only model files from huggingface.co/xautonomics are supported by dreamcatcher. Other GGUFs, including libraries you slice yourself, may load but are unsupported. Do not rename the files: the loader expects blk-NNNNN.gguf / parts-*.gguf / manifest.json exactly.

Layout

blk-00000.gguf … blk-00041.gguf   one transformer block each (blocks 0 and 1 are the dense blocks)
parts-embd.gguf                          token embedding (every stage loads it)
parts-output.gguf                        output norm + head (every stage loads it)
parts-other.gguf                         remaining non-block tensors, here `token_embd_norm.weight` (every stage loads it)
manifest.json                            file list, sizes, per-file + per-tensor hashes, source provenance

A stage that serves layers [A, B) needs blk-A … blk-(B-1) plus the parts-* files and manifest.json. Each file keeps the full source metadata (tokenizer included), and carries its own contribution to inkling.block_count / inkling.dense_block_count, which the loader sums over the assembled window.

Use

Single process (the whole model on one host):

llama-server --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF

Multi-host ring (stages exchange hidden states over TCP; each host needs low-latency, direct node-to-node links — see the dreamcatcher README):

llama-stage-runner --role tail --listen 8081 --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF --layers 21,42
llama-stage-runner --role head --connect <tail-host>:8081 --model-dir /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF --layers 0,21

Plan a split for your hosts (weights-only screen; the proof is a survived prefill), and verify the download before launching:

python3 -m layer_distribution.plan /models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF/manifest.json --node a:16GiB:200GiB:8080 --node b:24GiB:200GiB:8081
python3 -c "from layer_distribution import verify; r = verify('/models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF', '/models/Inkling-Small-UD-Q4_K_M.LAYR.GGUF/manifest.json'); print(r.passed, r.hash_verified)"

Provenance and license

Weights are byte-for-byte the tensors of unsloth/Inkling-Small-GGUF at revision 1a19ef82 (Unsloth's dynamic quantization of thinkingmachines/Inkling-Small). The five source shards were checked against the Hub's LFS SHA-256 digests before slicing; their names and sizes are recorded in manifest.json (source.shards[], source.content_hash). License follows the base model: Apache-2.0, see LICENSE. Credit to Thinking Machines Lab for the model and to Unsloth for the quantization.

About

Published by xAutonomics for dreamcatcher.