---

license: other

license_name: kimi-k3

license_link: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE

base_model: moonshotai/Kimi-K3

base_model_relation: quantized

library_name: llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• kimi
• kimi-k3
• moe
• mxfp4
• iq1_s

---

Author Note

**The author does not own enough hardware to run this 528.0293 GiB model.

Runtime results below were contributed by independent users and have not been

reproduced by the author. The reports predate the current top32/refit2 payload,

so their exact revision scope is stated explicitly.**

Kimi-K3 GGUF — IQ1_S routed experts / Q4_K eligible dense weights / F16–F32 structural tensors

Sub-2-bit GGUF conversion of the text model of

moonshotai/Kimi-K3 — a 2.8T-parameter

MoE (896 experts, 16 active) shipped in MXFP4 via quantization-aware training.

| | |

|---|---|

| Source | 1,453.8 GiB (MXFP4, 4.25 bpw), revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 |

| Output | 528.0293 GiB across 94 GGUF parts |

| Effective rate | 1.6319 bpw over 2,779,483,135,584 parameters |

| vs. source | 0.363× |

| Tensors | 2,573 |

| Architecture string | kimi-k3 |

---

⚠️ Read this before downloading 528 GiB

This does not load with any released version of llama.cpp. Kimi-K3 support is

still an open pull request:

ggml-org/llama.cpp#26185.

You must build from that branch (pwilkin/llama.cpp:kimi-k3-text).

**Hugging Face's autogenerated "Use this model" snippets (vLLM, Ollama, stock

llama.cpp) are not valid for this repository.** Ignore them.

Evidence levels — please do not conflate these

| Level | Status |

|---|---|

| Structural conformance of the current published files | Verified by the author |

| Quantizer byte-format correctness | Verified by the author against upstream gguf-py |

| Chat input formatting vs the official renderer | Verified by the author: 28/28 byte-identical; tools/schema also rendered directly with Minja |

| Full load and generation of the 94-part IQ1_S family | Reported by multiple independent users in Discussion #1 |

| Current top32/refit2 payload (promoted at 9dc1f8386c8d…, current repo revision 779f7cc92cc5…) run end to end | Not directly verified. It was promoted after the reports below. |

| Perplexity, standardized benchmark, source/Q2 output equivalence | Not measured. |

Independent runtime reports

Discussion #1

contains at least two independent full-generation reports:

• One user reported local llama.cpp generation at about 19.7 tokens/s and

supplied a screenshot, but did not state hardware, command line, llama.cpp

commit, or repository revision. Treat the speed as an observation, not a

reproducible benchmark.

• A second user reported running on 512 GB DDR5-4800, an Intel QYFS Sapphire

Rapids ES CPU, and 96 GB VRAM (1×4090 + 3×3090). They reported about

20 tokens/s prompt processing, 5 tokens/s generation, a 32K context,

and completion of a long one-shot HTML generation. The launch command and exact

offload split were not provided.

These reports establish that the 94-part IQ1_S layout and an earlier IQ1_S

payload could be fully loaded and used for generation, including on a large-RAM

consumer-GPU system. They remain third-party reports and were not reproduced by

the author.

Revision boundary: the reports were posted/updated on 2026-07-28 UTC. The

current top32/refit2 payload was promoted on 2026-07-29 at 01:52 UTC, so the

reports cannot be attributed to its exact bytes. The promotion retained the same

94-part tensor contract, tensor types, offsets, metadata, and total size while

replacing only IQ1_S expert payload bytes; sampled reconstruction metrics improved.

That makes the reports strong compatibility evidence, but not a direct current-

revision runtime or quality validation.

The second user noted that they needed to fix the chat template. Their exact

revision and failure mode were not supplied. The current card therefore does not

infer that the present 15,053-byte input template is broken: it is separately

28/28 byte-matched against K3 and rendered through Minja. K3 output parsing is a

distinct runtime concern on branches without a dedicated parser.

What was verified, on the actual uploaded files

Read back with HTTP range requests over all 94 parts (headers and KV only):

• All 94 parts present, split.no = 0..93, split.count = 94 everywhere
• split.tensors.count = 2573, written as INT32, identical in every part
• Sum of per-part tensor counts = 2573 — the value llama_model_loader

compares against weights_map.size()

• Tensor names identical to the set generated by the PR's create_tensor

calls: 0 missing, 0 extra, 0 duplicates

• Longest tensor name 29 chars (GGML_MAX_NAME is 64)
• Part 1 carries 66 KV entries: all 25 hparams the PR's loader reads, plus the

full vocabulary

• Type distribution: IQ1_S 276 / Q4_K 1067 / F32 1112 / F16 117 / Q8_0 1

manifest.json in this repository lists per-file size and SHA-256 for all 94 parts.

Known limitations and unverified assumptions

• No importance matrix. llama-quant.cpp marks IQ1_S as requiring one and

refuses to produce it without ("The result will be garbage, so bailing out").

The runtime reports show that collecting activation statistics is now possible

in principle, but no imatrix/calibration run was used for this artifact. This

remains a real quality limitation.

• The strongest tested codebook search is applied: stable top-32 exact-objective

candidates with two scale-refit rounds (see Codebook search below).

• Metal / Vulkan. The cross-layer residual uses ggml_dsv4_hc_pre, which has

CPU and CUDA implementations only. Other backends are expected to take the

scheduler's per-node fallback path; the graph contains 187 such nodes, so decode

would incur many device round trips. The reports above establish generation for

an earlier payload on llama.cpp setups that included GPU acceleration; they do

not verify the current top32/refit2 bytes, Metal, Vulkan, or other backends.

• Chat template: normal chat, thinking_effort, tool declarations/calls/results,

and response_format / response_schema are covered. Images and batched

conversations remain untested. See the tokenizer section.

---

Quantization mix

Routed experts hold 97.9% of the parameters, so they set the file size.

| Tensors | ggml type | bpw | Count | Size |

|---|---|---|---|---|

| ffn_{gate,up,down}_exps (routed experts) | IQ1_S | 1.5625 | 276 | 495.26 GiB |

| 2-D weights: attention, shared experts, latent MoE, dense MLP, token_embd | Q4_K | 4.5 | 1,067 | 28.58 GiB |

| output.weight | Q8_0 | 8.5 | 1 | 1.16 GiB |

| Norms, biases, router (ffn_gate_inp), conv1d, ssm_a | F32 | 32 | 1,112 | 2.25 GiB |

| ssm_f_b, attn_k_b, attn_v_b (row length not a multiple of 256) | F16 | 16 | 117 | 0.76 GiB |

The policy follows upstream's own rules in llama-quant.cpp: names not ending in

weight are never quantized, nor are *_norm.weight, ffn_gate_inp.weight, or

ssm_conv1d*. Row lengths were checked against block size for all 2,573 tensors.

---

Weight-space reconstruction error — not end-to-end model quality

**These measurements do not establish generation quality, perplexity, logit

fidelity, or routing fidelity.** They compare quantized tensors against the MXFP4

source in weight space only. The source is already 4-bit with 21 distinct values

per group, so this is a 4-bit → 1.5-bit requantization.

Full methodology and distributions are in quantization-report.json.

Format comparison on one tensor

layers.48.block_sparse_moe.experts.0.w1, all 11,010,048 elements,

rel_rmse = sqrt(mean((Q-W)²))/std(W), cos = dot(W,Q)/(‖W‖‖Q‖):

| Format | bpw | rel_rmse | cos |

|---|---|---|---|

| Q4_K (used for non-experts) | 4.5000 | 0.0881 | 0.996237 |

| Q2_K (previous artifact) | 2.6250 | 0.3313 | 0.951433 |

| IQ1_S (chosen, current top32/refit2 files) | 1.5625 | 0.455813 | 0.890112 |

| IQ1_S (top-8 + one refit, superseded) | 1.5625 | 0.4625 | 0.886748 |

| IQ1_S (old ternary-space snap, superseded) | 1.5625 | 0.5375 | 0.848225 |

| IQ1_M † | 1.7500 | 0.5696 | 0.850800 |

| TQ1_0 † | 1.6875 | 0.8017 | 0.790200 |

† IQ1_M and TQ1_0 were measured with the older codebook selection and were

not re-measured after the fix. Their numbers are therefore not directly

comparable to the current IQ1_S row; both would improve by an unknown amount.

Format choice. Under the old selection, TQ1_0 was Pareto-dominated by

IQ1_S (larger and worse) while IQ1_M was not dominated — it traded about 12%

more expert storage (~587 GiB total, +59 GiB) for slightly lower error. With stable top-32 selection and two refits, IQ1_S at 1.5625 bpw reaches

0.890112 on the representative tensor, better than the superseded top-8 result

without changing a single file byte in size. A fixed IQ1_M

would presumably move ahead again on error; it was not built, because the

objective was the smallest operating point around 530 GiB. This is stated as a

size/error trade-off, not as domination.

Distribution across layers

Sampled during conversion: one tensor per MoE layer

(block_sparse_moe.experts.0.w1), all elements measured, 92 of 92 MoE layers.

| Metric | mean | median | p90 | p99 | max | mean 95% CI (bootstrap) |

|---|---|---|---|---|---|---|

| rel_rmse | 0.455936 | 0.455958 | 0.456049 | 0.456150 | 0.456195 | [0.455913, 0.455958] |

| cos | 0.890050 | 0.890039 | 0.890124 | 0.890184 | min 0.889919 | [0.890039, 0.890062] |

The spread genuinely is this narrow — the tails are reported so this can be

checked rather than assumed.

Sampling gaps, stated explicitly: only w1 was measured, only expert index 0,

i.e. 1 of 2,688 expert tensors per layer (0.037%). w2 (input axis 3072) and w3

were not measured in production. Tail behaviour across experts within a layer is

unmeasured. With 1.5-bit weights, a handful of outlier tensors could matter, and

this sampling would not see them.

Codebook search: what was fixed, and what is still left

The codebook search minimises Σ(dl·(g+δ) − x)². Substituting u = x/dl − δ

gives dl²·Σ(g−u)², so candidates must be ranked against the unrounded u.

The original artifact used a top-8 shortlist and one scale refit. The current

artifact uses a stable top-32 shortlist ranked by the exact objective, preserves

the exact in-grid skip, and performs two scale-refit rounds.

Measured on the representative tensor:

| Grid selection | rel_rmse | cos |

|---|---|---|

| Old: ternary-space snap (superseded) | 0.5375 | 0.848225 |

| top-8 + one refit (superseded) | 0.4625 | 0.886748 |

| top-16 + one refit | 0.4582 | 0.888967 |

| top-32 + one refit | 0.4564 | 0.889879 |

| Current files: top-32 + two refits | 0.455813 | 0.890112 |

**The current files reduce mean reconstruction error by 15.2% at no change in

file size** (0.5375 → 0.455936 across 92 layers). Two properties make this efficient:

• clip(round(u)) is the exact nearest point of the lattice {-1,0,1}⁸, and the

grid is a subset of that lattice. So when the rounded code is one of the 2048

grid codes, the old snap was already optimal and no search is needed. That

is about 48% of groups, skipped losslessly.

• Refitting the scale after choosing the codebook entry is cheap relative to the

candidate evaluation. The current files use two refit rounds after stable top-32

selection and reach 0.890112 cosine on the representative tensor.

The current artifact uses the strongest tested 1.5625-bpw operating point. It still does not reach Q2_K's 0.951 cosine; this remains an aggressive 1.5-bit format.

An importance-matrix proxy that was tried and rejected

Since an imatrix requires running the model, an analytic substitute was derived:

for a tensor fed directly by an RMSNorm, E[x_i²] ∝ w_norm_i². The routed experts

are not fed by a norm — the reference implementation applies

routed_expert_norm to the expert output — so the importance was propagated

as v[j] = Σ_i W_down[j,i]² · ffn_norm_i². The result is nearly flat

(p99/p1 = 1.1×), because summing 7168 positive terms concentrates, and weighting

by it produced no improvement.

**This shows the proxy is unusable, not that importance weighting would not

help.** The diagonal approximation discards exactly the anisotropy being sought

(the off-diagonal of the input covariance).

---

Quantizer verification

llama.cpp cannot yet read this architecture, so llama-quantize could not be

used. The published expert payloads were generated by the fused stable-top-32

implementation and cross-checked against the authoritative NumPy path and

gguf-py 0.17.1 dequantization:

| Check | Result |

|---|---|

| TQ1_0: our bytes vs gguf.quants.quantize | 2,322,432 B identical |

| Q4_K / IQ1_S / IQ1_M: our bytes → upstream dequantize vs ours | max diff 0.0 |

| iq1s_grid: ggml-common.h uint64 table vs gguf-py 2-bit hex table | 0 mismatches |

| Q4_K 6-bit scale packing: get_scale_min_k4 inverse round trip | exact |

| MXFP4 source repack (reference dequant vs GGML dequant) | 0 mismatches / 11,010,048 elements |

Two details worth recording:

• GGML rounds with lroundf (away from zero); NumPy's np.rint is banker's

rounding. Every MXFP4 value is a dyadic rational, so x/scale lands exactly on

.5 often and the difference changes output bytes. TQ1_0 byte equality only

appeared after matching this.

• A naive IQ1_S search scans 2048 codebook entries per 8 elements. Since a ternary

8-tuple has only 3⁸ = 6561 forms, a one-time "6561 → nearest grid index" table

makes it O(1) per group. The same table also supplies the ranked candidate list

used by the fixed search, and marks which codes need no search at all.

---

Conversion details

The file records what was done to it:

kimi-k3.conversion.contract         = llama.cpp PR #26185 (pwilkin/kimi-k3-text)
kimi-k3.conversion.source_revision  = 9f62e4e9fffbd0a83ddd60e1c209d828994b3569
kimi-k3.conversion.source_quant     = compressed-tensors/mxfp4-pack-quantized
kimi-k3.conversion.expert_quant     = IQ1_S
kimi-k3.conversion.dense_quant      = Q4_K
kimi-k3.conversion.a_log_transform  = -exp(A_log[:n_head])
kimi-k3.conversion.attn_res_fused   = res_norm*res_proj[0] in float32
kimi-k3.conversion.kv_b_split       = k_b(transposed)+v_b
kimi-k3.conversion.expert_stack_dim = 0
kimi-k3.conversion.vision_excluded  = true
kimi-k3.conversion.imatrix          = false
kimi-k3.conversion.defaults_used    = rope_theta

Notable transforms:

• ssm_a = -exp(A_log[:n_head]). K3 stores A_log with shape [head_dim]

= 128, but only the first num_heads = 96 entries are used and the loader

expects 96. Without the narrowing:

check_tensor_dims: tensor 'blk.0.ssm_a' has wrong shape; expected 96, got 128.

The model's own bundled modeling_kimi_linear.py declares A_log with

num_heads elements while the shipped weight has head_dim; the weight is

authoritative and only its first 96 entries matter.

• AttnRes is fused. <x>_res_norm.weight * <x>_res_proj.weight[0] folded into

one float32 vector per site (attn_res_score, ffn_res_score,

output_res_score). Exact, not an approximation: the reference

_apply_attn_res() computes the same product itself. Removes

2 × 93 + 1 = 187 tensors (2,760 → 2,573).

• kv_b_proj split into attn_k_b (transposed) and attn_v_b, the path the

loader requires when the unsplit tensor is absent.

• rope_theta is absent from config.json; 10000.0 comes from the class

default in configuration_kimi_k3.py, recorded in defaults_used rather than

silently assumed.

Not included

The vision tower is excluded (MoonViT-3D, 27 blocks, plus mm_projector;

168 tensors, ~0.83 GiB). PR #26185 covers the text model only, and

llama_model_loader::done_getting_tensors() runs with partial = false, so any

tensor the architecture does not create makes the load fail outright. K3's MoonViT

also differs from the KIMIVL tower already in clip.cpp (wqkv,

patch_embed.pos_emb [64,64,1024]).

Tokenizer and chat template

K3 ships neither tokenizer.json nor tokenization_kimi.py — only

encoding_k3.py and tiktoken.model. The vocabulary was rebuilt directly from

tiktoken.model (sha256 identical to K2's):

• tokenizer.ggml.model = gpt2, tokenizer.ggml.pre = kimi-k2
• 163,840 tokens, 163,328 merges, 256 single-byte tokens, 0 unused slots
• pre-tokenizer checksum 81212dc7… matches the value llama.cpp already knows
• BOS 163584 [BOS], EOS 163586 <|end_of_msg|> (chat-turn terminator from

generation_config.json, not 163585 [EOT]/[EOS] which ends a document),

EOT 163593

• add_bos_token = false — the reference renderer does not emit BOS, so adding

one would double it

A chat template IS embedded, and it was differentially tested

K3 has no static Jinja template; encoding_k3.py builds an XTML conversation in

Python. Correct vocabulary alone does not reproduce the training-time format,

so a Jinja equivalent is embedded as tokenizer.chat_template.

The format, read out of encoding_k3.py:

<|open|>message role="user"<|sep|>hello<|close|>message<|sep|><|end_of_msg|>
<|open|>message role="assistant"<|sep|><|open|>think<|sep|>      <- generation prompt

with _open_tag(tag, attrs) = <|open|> + tag + k="v"… + <|sep|>,

_close_tag(tag) = <|close|> + tag + <|sep|>, and attribute values escaped

as &→&, "→".

**Differential test against the reference implementation: 28/28 cases

byte-identical.** The cases cover standard/multi-turn chat, CJK and attribute

escaping, thinking_effort, tool declarations, assistant tool calls, ordered

tool results, and nested response_schema values.

The template extracted from the published GGUF was additionally rendered by

llama.cpp's Minja engine (commit 91f8c9c5). The tools/tool-result case (1,218

bytes) and nested-schema case (621 bytes) were both byte-identical to K3's own

renderer. This caught and fixed three Minja-specific issues: dict-literal

namespace({...}), unsupported tojson(sort_keys=true), and an items field

colliding with the namespace object's method. Images and batched conversations

remain untested.

Provenance and independent conformance. The file in this artifact began with

Xenova's port and was expanded

and patched locally; it is not claimed to be byte-identical to the separate

upstream implementation in Moonshot PR #66.

PR #66 and ChatLint's K3 findings

provide independent, oracle-backed conformance work. The Minja

namespace(items=[]) collision reproduced here was accepted and fixed upstream

in ChatLint commit ecb1a727

and on the PR branch. Against this artifact's 15,053-byte file, ChatLint pinned to

that commit reports 294/294 checks, 0 errors, 0 warnings. ChatLint renders with

transformers-style Jinja2 rather than Minja and checks structural properties, so

it supplements rather than replaces the 28/28 oracle comparison and direct Minja

runs above.

One limitation is shared with PR #66: sandboxed Jinja cannot parse a JSON string

inside tool_calls[].function.arguments. This template emits a valid

<|open|>json type="object" fallback for a non-empty string; callers that require

the reference renderer's per-argument XTML must parse the string into an object

before applying the template.

The exact same 15,053-byte template is embedded in part 1 and published separately

as chat_template.jinja. Reproduce the differential test with

tools/verify_chat_template.py in the conversion repository.

Files

• 94 parts, Kimi-K3-IQ1_S-000NN-of-00094.gguf. Pass part 1 to llama.cpp; the

rest are discovered automatically.

• manifest.json — per-file size and SHA-256, tensor totals, type counts.
• quantization-report.json — error methodology, definitions, sampling coverage

and gaps, distributions with bootstrap CI, and the selected operating point.

• chat_template_verify_minja.json — the 28-case differential test against

encoding_k3.py.

• tokenizer.chat_template is embedded in part 1 (15,053 UTF-8 bytes), and the

same file is available as chat_template.jinja.

License

Inherits the Kimi K3 License.

Redistribution of derivative works is permitted. Commercial "Model as a Service"

use above the revenue thresholds in the license requires a separate agreement

with Moonshot AI.