---

license: other

license_name: modified-mit

base_model: moonshotai/Kimi-K3

library_name: gguf

pipeline_tag: text-generation

tags:

- gguf

- kimi-k3

- moe

- quantization

- iq1_s

- llama.cpp

- sparkinfer

- blackwell

---

Kimi-K3 UD-IQ1_S GGUF

Kimi-K3 (2.78 T parameters) compressed to 330.2 GB IQ1_S GGUF (307.49 GiB, 9 shards) — about 21% of the ~1.56 TB source artifact.

Every one of the 82,432 routed experts is present — no experts are dropped. The GGUF loads without model-specific patches on an existing Kimi-K3-capable llama.cpp build.

> The exact method used to create this model will be released soon. This card documents what the model is, how to run it, and how it measures — not how it was built.

Benchmarks

| Benchmark | K3 IQ1_S GGUF | Full K3 | Notes |

|---|---|---|---|

| HumanEval | 94.5% (155/164) | 94.5% | 1:1 match |

| MMLU | 79.49% | 85% | Within expected IQ1_S gap |

| AIME | 92.5% | 96% | |

| GSM8K | 95% | 95% | |

| Metric | Value |

|---|---|

| Size | 330,167,807,328 bytes / 307.49 GiB (9 shards) |

| vs. smallest published K3 quant | 44% smaller (594 GB → 330 GB) |

| Weights resident | 316,036 MiB, measured on 8× A100-80GB |

| Speed (llama.cpp, single-stream) | 10.8 tok/s |

| Speed (3× DGX Spark TP3, SparkInfer) | 12.5492 tok/s (K=8 profile, 112/112 accepted drafts) |

| WikiText-2 PPL | 5.3610 ± 0.02917 (canonical wiki.test.raw, all 568 chunks, -c 512) |

| Full-MXFP4 top-1 agreement | 73.741 ± 0.243% (32-window mixed-domain paired protocol) |

| Context | Architecture limit 1,048,576; 3× Spark TP3 practical ceiling ~8K for full resident weights (~113 GiB/rank) |

---

1. SparkInfer TP3/TP4 — Optimal Path (Blackwell)

Current measured throughput: 12.5468 and 12.5516 tok/s, averaging 12.5492 tok/s around a matched 6.4930 baseline, with 112/112 accepted drafts and identical token IDs. This is the K=8 repeat-heavy structured profile from the SparkInfer TP3 recipe, verified on real DGX Spark (GB10) hardware.

SparkInfer is optimized for all Blackwell chips (H200, GB10/DGX Spark, and other GB10-based systems). The gittensor-ai-lab/sparkinfer-k3 repo runs on 8× H200 as the reference configuration; DGX Spark (GB10) has a reduced footprint (~113 GiB/rank under 121 GiB).

| Runtime | Hardware | Measured decode |

|---|---|---|

| SparkInfer TP3 | 3× DGX Spark (GB10) | 12.5492 tok/s (K=8 profile, 112/112 drafts) |

| SparkInfer TP4 | 4× DGX Spark (GB10) | 7.90 tok/s |

| llama.cpp | 8× A100-80GB | 10.8 tok/s |

| vLLM TP3 + DSpark | 3× H200 | 52.454 tok/s (coding) |

Why MY SparkInfer Patch Is Optimal

• Correctness verified 2026-08-11: The distributed (TP3/TP4) generation path had a bug in kimi_k3_dist_rank.cpp: it never set the GGUF capability flags that gate loading, so shared experts, routed-expert normalization, and MLA attention gate were silently skipped on every token — real weights present, never computed. MY fix adds the same capability probe every other code path already had.
• Footprint: ~113 GiB/rank fits comfortably under GB10's 121 GiB.
• Current measurements (corrected engine):

- Load time: ~5m45s/rank (was ~30-60 min pre-patch 0013)

- SPARKINFER_K3_KDA_FUSE=0: +20% over MY own baseline

- K=4 speculative decoding candidate: 10.3002 and 11.3231 tok/s (10.8117 avg), +68.2% over matched 6.4272 baseline, 95/95 accepted drafts

- K=8 repeat-heavy profile: 12.5468 and 12.5516 tok/s (12.5492 avg) around 6.4930 baseline, 112/112 accepted drafts, identical token IDs

• Not valid for non-Blackwell chips: Tested on physical fleets, hard memory ceiling on other hardware.

Full launch procedure (see kimi-k3-neuron-tp3-dgxspark-recipe):

1. 3 or 4 DGX Spark (GB10) nodes, each with ≥320 GB free NVMe
2. Pull model with hf_xet transport
3. Apply verified 28-patch chain through 0026
4. Build SparkInfer with MY patch series
5. Copy built dist/ tree to every rank node
6. Launch rank 0 (coordinator) with sparkinfer --model ./Kimi-K3-UD-IQ1_S --tp 3 --rank 0 --listen --ctx-len 8192 --jdba-band
7. Launch rank 1 and rank 2 (workers) with --coord <rank0-host>:9000 --jdba-band
8. Verify all ranks connected, then query

Shards load from Kimi-K3-UD-IQ1_S-00001-of-00009.gguf (all 9 parts required).

Speculative Decoding (Experimental, Opt-In)

Patches 0021–0026 add n-gram/prompt-lookup speculative decoding, match-confidence gating, recursive-majority continuation, and distributed LM-head banding. On real TP3 hardware, 64-token generations show:

• Code/structured: +27% with --spec-draft 4
• Literal repetition: +83% with --spec-draft 4
• Freeform prose: −3.7%
• Mean across all three: +34.5%

---

2. llama.cpp — Fallback (Any Chip)

Requires a llama.cpp build with Kimi-K3 support — upstream PR #26185 or the Unsloth K3 fork.

llama-server \
  -m Kimi-K3-UD-IQ1_S-00001-of-00009.gguf \
  -ngl 99 --jinja \
  -c 8192 --parallel 1 \
  -ctk f16 -ctv f16 -fa on \
  --temp 1.0 --top-p 0.95

---

3. vLLM TP3 + DSpark — Last Resort (Non-Blackwell)

| Runtime | Hardware | Measured decode | Recipe |

|---|---|---|---|

| vLLM TP3 + DSpark | 3× H200 | 52.454 tok/s (coding) | kimi-k3-neuron-tp3-vllm-recipe |

---

Download

This repository is Xet-backed. Use the hf_xet transport — chunk-deduplicated and parallel:

pip install -U "huggingface_hub[hf_xet]"
hf auth login

export HF_XET_HIGH_PERFORMANCE=1
hf download vcruz305/Kimi-K3-GGUF --local-dir ./Kimi-K3-UD-IQ1_S

Model shards and template only:

hf download vcruz305/Kimi-K3-GGUF --local-dir ./Kimi-K3-UD-IQ1_S \
   --include "Kimi-K3-UD-IQ1_S-*.gguf" --include "k3_chat_template.jinja"

All 9 shards are required; llama.cpp loads them automatically from Kimi-K3-UD-IQ1_S-00001-of-00009.gguf.

Two things learned moving this artifact around:

• Xet dedup is per chunk, not per file — when shard 00001 was rewritten to embed the chat template, the 39.4 GB re-upload transferred 395 kB.
• Advertised bandwidth is meaningless: a host advertising 538 Mbps sustained 7,828 Mbps on real Xet transfers.

---

Status

Experimental, and it works. It loads, reasons in K3's native format, and scores 94.5% on HumanEval — measured on the full 164 problems, not a subset.

---

Changelog

• 2026-08-12: Renamed GGUF shards to Kimi-K3-UD-IQ1_S-*-of-00009.gguf (server-side pointer rename; no weight re-upload).
• 2026-08-11: Correctness fix applied and verified in SparkInfer TP3 engine — kimi_k3_dist_rank.cpp now sets GGUF capability flags that gate loading of shared experts, routed-expert normalization, and MLA attention gate. Pre-fix measurements were invalid (engine skipped real computation).
• Pre-2026-08-11: Historical speed tables deprecated — measured on engine skipping computation.
• Initial release: 330 GB IQ1_S GGUF with all 82,432 experts present, verified loading on Unsloth llama.cpp fork.