L-Alchemyst/Step-5-Preview-GGUF overview
<style .qz wrap { background: 0d1117; color: e6edf3; font family: 'Inter', system ui, apple system, sans serif; font size: 1rem; line height: 1.7; padding: 2re…
Runs locally from ~2.56 GB disk (4 GB VRAM class GPUs with llama.cpp / guIDE).
Repository Files & Downloads
| File | Type | Quantization | Size | Link |
|---|---|---|---|---|
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00001-of-00030.gguf | GGUF | IQ3_KS | 8.74 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00002-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00003-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00004-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00005-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00006-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00007-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00008-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00009-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00010-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00011-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00012-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00013-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00014-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00015-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00016-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00017-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00018-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00019-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00020-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00021-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00022-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00023-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00024-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00025-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00026-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00027-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00028-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00029-of-00030.gguf | GGUF | IQ3_KS | 8.61 GB | Download |
| IQ3_KS-mix/step5-preview-IQ3_KS-mix-00030-of-00030.gguf | GGUF | IQ3_KS | 6.39 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00001-of-00036.gguf | GGUF | IQ4_KS | 9.20 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00002-of-00036.gguf | GGUF | IQ4_KS | 9.28 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00003-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00004-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00005-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00006-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00007-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00008-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00009-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00010-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00011-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00012-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00013-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00014-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00015-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00016-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00017-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00018-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00019-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00020-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00021-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00022-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00023-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00024-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00025-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00026-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00027-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00028-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00029-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00030-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00031-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00032-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00033-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00034-of-00036.gguf | GGUF | IQ4_KS | 8.18 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00035-of-00036.gguf | GGUF | IQ4_KS | 9.23 GB | Download |
| IQ4_KS-mix/step5-preview-IQ4_KS-mix-00036-of-00036.gguf | GGUF | IQ4_KS | 3.86 GB | Download |
| mmproj-step5-vision-bf16.gguf | GGUF | BF16 | 4.10 GB | Download |
| mmproj-step5-vision-q8_0.gguf | GGUF | Q8_0 | 2.56 GB | Download |
Model Details
| Model ID | L-Alchemyst/Step-5-Preview-GGUF |
|---|---|
| Author | L-Alchemyst |
| Pipeline | — |
| License | other |
| Base model | — |
| Last modified | 2026-10-06T04:17:43.000Z |
Model README
---
license: other
---
<style>
.qz-wrap {
background: #0d1117;
color: #e6edf3;
font-family: 'Inter', system-ui, -apple-system, sans-serif;
font-size: 1rem;
line-height: 1.7;
padding: 2rem 2.5rem;
border-radius: 8px;
margin-bottom: 2rem;
}
.qz-wrap * { color: #e6edf3; }
.qz-wrap a { color: #58a6ff; text-decoration: none; }
.qz-wrap a:hover { text-decoration: underline; }
.qz-header {
font-size: 2.2rem;
font-weight: 700;
color: #ffffff;
border-bottom: 3px solid #22d3ee;
padding-bottom: 0.5rem;
margin-bottom: 1rem;
}
.qz-subtitle {
font-size: 1.05rem;
color: #8b949e;
margin-top: 0.2rem;
margin-bottom: 0.4rem;
}
.qz-section {
font-size: 1.3rem;
font-weight: 600;
color: #ffffff;
margin-top: 2rem;
margin-bottom: 0.6rem;
}
.qz-sub {
font-size: 1.1rem;
font-weight: 600;
color: #7ab8d4;
margin-top: 1.4rem;
margin-bottom: 0.5rem;
}
.qz-wrap table.qz-table {
display: table;
width: 100%;
min-width: 100%;
max-width: 100%;
table-layout: fixed;
border-collapse: collapse;
border-spacing: 0;
margin-bottom: 1rem;
}
.qz-wrap table.qz-table th,
.qz-wrap table.qz-table td {
padding: 0.55rem 1rem;
text-align: left;
border-bottom: 1px solid #30363d;
box-sizing: border-box;
}
.qz-wrap table.qz-table th:first-child,
.qz-wrap table.qz-table td:first-child {
width: 30%;
}
.qz-wrap table.qz-table th:last-child,
.qz-wrap table.qz-table td:last-child {
width: 70%;
}
.qz-wrap table.qz-table th {
background: rgba(34, 211, 238, 0.08);
color: #22d3ee;
font-weight: 700;
text-transform: uppercase;
font-size: 0.72rem;
letter-spacing: 0.06em;
}
.qz-wrap table.qz-table tr:nth-child(even) td {
background: rgba(255, 255, 255, 0.02);
}
.qz-note {
font-size: 0.88rem;
color: #8b949e;
font-style: italic;
}
.qz-status {
font-size: 0.9rem;
color: #d69e2e;
font-weight: 500;
padding: 0.6rem 1rem;
border: 1px solid #d69e2e;
border-radius: 4px;
display: inline-block;
margin-bottom: 1rem;
}
.qz-recipe-toggle {
margin: 0.8rem 0 1rem 0;
border: 1px solid #0891b2;
border-radius: 6px;
overflow: hidden;
width: 100%;
}
.qz-recipe-toggle summary {
font-size: 0.9rem;
font-weight: 600;
color: #22d3ee;
padding: 0.6rem 1rem;
cursor: pointer;
user-select: none;
background: rgba(34, 211, 238, 0.06);
list-style: none;
}
.qz-recipe-toggle summary::before { content: '▸ '; }
.qz-recipe-toggle[open] summary::before { content: '▾ '; }
.qz-recipe-toggle summary:hover { background: rgba(34, 211, 238, 0.12); }
.qz-recipe-body {
font-family: 'JetBrains Mono', 'Fira Code', monospace;
font-size: 0.8rem;
padding: 0.8rem 1rem;
line-height: 1.7;
background: rgba(34, 211, 238, 0.03);
color: #e6edf3;
}
.qz-recipe-body .rq-section {
color: #22d3ee;
font-weight: 700;
margin-top: 0.6rem;
}
.qz-recipe-body .rq-line { display: block; }
.qz-recipe-body .rq-quant {
color: #67e8f9;
font-weight: 600;
}
.qz-code {
font-family: 'JetBrains Mono', 'Fira Code', monospace;
font-size: 0.82rem;
display: block;
padding: 1rem;
margin: 0.8rem 0;
overflow-x: auto;
border-left: 3px solid #0891b2;
background: rgba(34, 211, 238, 0.04);
color: #e6edf3;
line-height: 1.7;
white-space: pre;
}
.qz-chart-container {
text-align: center;
margin: 0.5rem 0 1.5rem 0;
}
.qz-chart-container img {
border: 1px solid #30363d;
border-radius: 4px;
max-width: 100%;
height: auto;
}
.qz-credit {
font-size: 0.9rem;
color: #8b949e;
margin-top: 2rem;
padding-top: 1rem;
border-top: 1px solid #30363d;
}
.qz-credit a { color: #22d3ee; }
</style>
<html>
<head>
<meta charset="utf-8">
</head>
<body>
<div class="qz-wrap">
<div class="qz-header">Step-5-Preview</div>
<div class="qz-subtitle">Imatrix quants best-in-class for <a href="https://github.com/NF09830453/ik_llama.cpp/tree/step5-preview-dense">ik_llama.cpp (Step-5 fork)</a></div>
<div class="qz-subtitle">NOTE: ik_llama.cpp is highly versatile and runs mainline llama.cpp quants such as existing GGUFs from bartowski, unsloth, mradermacher, Aes Sedai etc</div>
<div class="qz-status">IQ3_KS-mix & IQ4_KS-mix released ✅ — more quants to come</div>
<div class="qz-section">General Specs</div>
<table class="qz-table">
<tr><th></th><th>Value</th></tr>
<tr><td>Architecture</td><td>Step-5 (MoE, 352 experts × 8 active + 1 shared, sparse GQA indexer)</td></tr>
<tr><td>Params</td><td>600.1B total / 27B active</td></tr>
<tr><td>Layers</td><td>92 (3 dense + 89 MoE) + 3 MTP</td></tr>
<tr><td>Attention</td><td>64 q heads / 4 kv, head_dim 192; sliding (512) × 3 → full, × 92</td></tr>
<tr><td>Context</td><td>1,048,576</td></tr>
<tr><td>Vocab</td><td>128,896</td></tr>
<tr><td>Imatrix</td><td>step-preview-A.imatrix, 400K tokens @ ctx 8192, ubergarm-v02 corpus, sparse-enabled runtime</td></tr>
</table>
<div class="qz-section">Imatrix</div>
<table class="qz-table">
<tr><th></th><th>Value</th></tr>
<tr><td>File</td><td>step-preview-A.imatrix (713 MB, 977/977 tensor entries)</td></tr>
<tr><td>Corpus</td><td>ubergarm-imatrix-calibration-corpus-v02</td></tr>
<tr><td>Tokens</td><td>400K (50 chunks x 8192 tokens)</td></tr>
<tr><td>Context</td><td>8192</td></tr>
<tr><td>Runtime</td><td>ik_llama.cpp with sparse indexer active (see flags below)</td></tr>
<tr><td>Coverage</td><td>977 entries; gaps are benign (gate_shexp kept at iq6_k, indexer.k q8_0 anyway, MTP/embeddings at standard tiers)</td></tr>
</table>
<div class="qz-note">The imatrix was collected with the sparse selection ACTIVE at deployment context, so activation statistics match how the model actually runs. Dense-mode calibration at ctx 8192 would measure the wrong distribution: with top-k 512 blocks the sparse path discards most keys at that length.</div>
<div class="qz-note">Download: <a href="https://huggingface.co/L-Alchemyst/Step-5-Preview-GGUF/blob/main/step-preview-A.imatrix">step-preview-A.imatrix</a> — reusable for any tier quant from the BF16 master.</div>
<div class="qz-section">About our inference code</div>
<div class="qz-note">The sparse GQA indexer implementation (SSMax, CSA block compression, sparse selection) ships with ik_llama.cpp and is being open-sourced. Everything was derived from the model's sparse_config metadata and validated empirically: exact needle retrieval, PPL A/B tests, and logit-equivalence gates against the dense baseline. The knobs below make every candidate switchable without re-quantizing: the quants themselves stay valid.</div>
<div class="qz-note"><strong>Seriously experimental:</strong> this is a from-scratch implementation, not a port of reference code. Multiple GPU configurations are untested, including <code>-sm tensor</code> and <code>-sm graph</code> (split modes). Known reports: <code>-sm graph</code> segfault and layer-parallel token duplication under dense multi-GPU configs (fixes pending; single-GPU and default split mode unaffected). The Quick Start command uses the default split mode, which is the only one exercised. Expect rough edges and please report what you find.</div>
<table class="qz-table">
<tr><th>Flag</th><th>Meaning</th></tr>
<tr><td>IK_SPARSE=1</td><td>Activates the sparse indexer on full-attention layers. Selection masks the attention instead of attending to every cached key. Default 0 runs dense (telemetry only).</td></tr>
<tr><td>IK_CSA=2</td><td>Block-compressed indexer keys via the learned z projection (csa_z_norm_type=none in sparse_config). Mode 1 pools cached keys by averaging; mode 0 is token-grain. The z-proj variant measured better PPL (-8.6% at 8k ctx) and exact needle retrieval.</td></tr>
<tr><td>IK_SSMAX=1</td><td>Per-query-head logit scaling (ssmax_s weights, ln(n) factor). PPL-neutral at deployment context and part of the config that passed needle retrieval exactly.</td></tr>
<tr><td>IK_SEL_SOFT=0.5</td><td>Soft-gate penalty (attention-logit units) applied to keys outside the indexer's kept set. <strong>Set this whenever IK_SPARSE=1</strong> — default 0 is the legacy hard cliff (strict top-k masking), which starves long-context recall. 0.5 is the validated setting; larger values approach dense-equivalent attention.</td></tr>
<tr><td>IK_SEL_KEEP_PCT</td><td>Optional kept-slot scaling override (default 0 = fixed 4096-slot floor, proven sufficient with the soft gate; 50 = keep half of context; 100 = effectively dense selection). Experimentation knob only.</td></tr>
</table>
<div class="qz-note"><strong>✅ Fixed — sparse long-context recall (update):</strong> early builds hard-masked every key past the kept-slot budget (4096 slots); beyond roughly 4–8k context that cliff starved verbatim recall — facts and paths came back as fabricated replacements while short tokens survived. Root cause was the hard rank cliff in the selection mask. The fix is a constant soft gate (<code>IK_SEL_SOFT=0.5</code>): keys outside the kept set keep a small attention-logit penalty instead of −∞, so strongly-scoring keys the indexer under-ranked still attend. Validated prod-parity at IQ3-tier stress mix (worst case): needle recall 18/18 at every depth 4.4k–44k, fresh AND multi-turn session, fabrications 0, n=3. Requires building current source (commit <code>dd4359b9</code> or newer — the Quick Start clone above already includes it). <strong>Update (commit <code>81ac5fc7</code>):</strong> the indexer now runs fused (selection scored and masked in-kernel) — behavior validated identical to the previous path, but compute-buffer memory drops ~70% at prod settings (≈2.2 GiB vs ≈7.4 GiB at 16k ctx), so long contexts fit in far less VRAM (launch-checked at 180k ctx). The old workaround <code>IK_SEL_KEEP_PCT=100</code> is obsolete; kept-slot scaling remains only as an override knob (default 0 = fixed 4096-slot floor, proven sufficient with the soft gate). Honest boundaries: recall validated up to 44k context (larger contexts launch — checked to 180k — but recall beyond 44k is untested, and the 1M max is untested); sparse selection is still mask-only — it shapes <em>which</em> keys attend but does not cut attention compute (expect a small decode tax vs dense, unchanged by the fused rewrite, until KV-row gather kernels land).</div>
<div class="qz-note">All flags are read once at process start and stay fixed for the run. The Quick Start exports the exact combination the quant was calibrated and validated with (<code>IK_SEL_SOFT=0.5</code> included).</div>
<div class="qz-section">PPL Analysis</div>
<div class="qz-note">TODO: PPL / KLD evaluation vs base reference pending.</div>
<div class="qz-section">Quant Collection</div>
<!-- IQ4_K -->
<div class="qz-sub">✅ IQ4_KS-mix <span class="qz-note">310.1 GiB · 36 shards</span></div>
<div class="qz-note">TODO: PPL / KLD evaluation vs base reference pending.</div>
<table class="qz-table">
<tr><th>Metric</th><th>Value</th></tr>
<tr><td>PPL (quant)</td><td>Pending</td></tr>
<tr><td>PPL (base ref)</td><td>Pending</td></tr>
<tr><td>(PPL(Q)/PPL(base)) - 1</td><td>Pending</td></tr>
<tr><td>KLD</td><td>Pending</td></tr>
<tr><td>Same top-p</td><td>Pending</td></tr>
<tr><td>Δp RMS</td><td>Pending</td></tr>
</table>
<details class="qz-recipe-toggle">
<summary>Quantization recipe</summary>
<div class="qz-recipe-body">
<span class="rq-section"># Attention (all layers)</span>
<span class="rq-line">blk\..*\.attn_q\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_k\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_v\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_output\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_gate\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-section"># Indexer (full-attn layers)</span>
<span class="rq-line">blk\..*\.indexer\.q\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.indexer\.k\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.indexer\.z\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.indexer\.w\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-section"># Dense layers 0-2 + 91</span>
<span class="rq-line">blk\.(0|1|2|91)\.ffn_down\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-line">blk\.(0|1|2|91)\.ffn_(gate|up)\.weight=<span class="rq-quant">iq5_ks</span></span>
<span class="rq-line">blk\.(0|1|2|91)\.ffn_gate\.weight=<span class="rq-quant">iq5_ks</span></span>
<span class="rq-section"># Shared experts</span>
<span class="rq-line">blk\..*\.ffn_down_shexp\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-line">blk\..*\.ffn_gate_shexp\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-line">blk\..*\.ffn_up_shexp\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-section"># Routed experts (uniform)</span>
<span class="rq-section"># down one tier above up/gate</span>
<span class="rq-line">blk\..*\.ffn_down_exps\.weight=<span class="rq-quant">iq4_k</span></span>
<span class="rq-line">blk\..*\.ffn_(gate|up)_exps\.weight=<span class="rq-quant">iq4_ks</span></span>
<span class="rq-section"># MTP / nextn tail</span>
<span class="rq-section"># shared_head_head = draft lm_head</span>
<span class="rq-section"># eh_proj = layer-input projector</span>
<span class="rq-line">blk\..*\.nextn\.eh_proj\.weight=<span class="rq-quant">iq4_kss</span></span>
<span class="rq-line">blk\..*\.nextn\.shared_head_head\.weight=<span class="rq-quant">iq5_ks</span></span>
<span class="rq-section"># Embeddings / output</span>
<span class="rq-line">token_embd\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-line">output\.weight=<span class="rq-quant">iq6_k</span></span>
</div>
</details>
<!-- IQ3_K -->
<div class="qz-sub">✅ IQ3_KS-mix <span class="qz-note">256.3 GiB · 30 shards</span></div>
<div class="qz-note">TODO: PPL / KLD evaluation vs base reference pending.</div>
<table class="qz-table">
<tr><th>Metric</th><th>Value</th></tr>
<tr><td>PPL (quant)</td><td>Pending</td></tr>
<tr><td>PPL (base ref)</td><td>Pending</td></tr>
<tr><td>(PPL(Q)/PPL(base)) - 1</td><td>Pending</td></tr>
<tr><td>KLD</td><td>Pending</td></tr>
<tr><td>Same top-p</td><td>Pending</td></tr>
<tr><td>Δp RMS</td><td>Pending</td></tr>
</table>
<details class="qz-recipe-toggle">
<summary>Quantization recipe</summary>
<div class="qz-recipe-body">
<span class="rq-section"># Attention (all layers)</span>
<span class="rq-line">blk\..*\.attn_q\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_k\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_v\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_output\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.attn_gate\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-section"># Indexer (full-attn layers)</span>
<span class="rq-line">blk\..*\.indexer\.q\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.indexer\.k\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.indexer\.z\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-line">blk\..*\.indexer\.w\.weight=<span class="rq-quant">q8_0</span></span>
<span class="rq-section"># Dense layers 0-2 + 91</span>
<span class="rq-line">blk\(0|1|2|91)\.ffn_down\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-line">blk\(0|1|2|91)\.ffn_(gate|up)\.weight=<span class="rq-quant">iq5_ks</span></span>
<span class="rq-section"># Shared experts</span>
<span class="rq-line">blk\..*\.ffn_down_shexp\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-line">blk\..*\.ffn_gate_shexp\.weight=<span class="rq-quant">iq5_ks</span></span>
<span class="rq-line">blk\..*\.ffn_up_shexp\.weight=<span class="rq-quant">iq5_ks</span></span>
<span class="rq-section"># Routed experts (uniform)</span>
<span class="rq-section"># down one tier above up/gate</span>
<span class="rq-line">blk\..*\.ffn_down_exps\.weight=<span class="rq-quant">iq4_ks</span></span>
<span class="rq-line">blk\..*\.ffn_(gate|up)_exps\.weight=<span class="rq-quant">iq3_ks</span></span>
<span class="rq-section"># MTP / nextn tail</span>
<span class="rq-section"># shared_head_head = draft lm_head</span>
<span class="rq-section"># eh_proj = layer-input projector</span>
<span class="rq-line">blk\..*\.nextn\.eh_proj\.weight=<span class="rq-quant">iq4_kss</span></span>
<span class="rq-line">blk\..*\.nextn\.shared_head_head\.weight=<span class="rq-quant">iq5_ks</span></span>
<span class="rq-section"># Embeddings / output</span>
<span class="rq-line">token_embd\.weight=<span class="rq-quant">iq6_k</span></span>
<span class="rq-line">output\.weight=<span class="rq-quant">iq6_k</span></span>
</div>
</details>
<div class="qz-section">Making your own quants (conversion pipeline)</div>
<div class="qz-note"><strong>IMPORTANT:</strong> the <code>convert_hf_to_gguf.py</code> converter in ik_llama.cpp is <strong>BROKEN</strong> for this architecture — do not use it. The correct pipeline is:</div>
<pre><code># 1. Convert HF -> BF16 GGUF with MAINLINE llama.cpp's converter + the Step-5 patch
(patch + instructions by avar6: https://huggingface.co/avar6/Step5-Preview)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && git apply step5.patch # patch from the avar6 repo above
python convert_hf_to_gguf.py /path/to/Step5-Preview-HF \
--outfile step5-preview-bf16.gguf --outtype bf16</code></pre>
<pre><code># 2. Indexer tensor surgery — the mainline converter output lacks the 184 sparse-indexer
tensors (1783 total after surgery). Insert them from the HF checkpoint:
script: https://huggingface.co/L-Alchemyst/Step-5-Preview-GGUF/blob/main/indexer_insert.py
python3 indexer_insert.py plan # dry-run: parse + assert + print the bill
python3 indexer_insert.py apply # perform the surgery (keeps a .bak)
python3 indexer_insert.py verify # reparse + byte-compare vs HF
(edit the CONFIG block at the top of the script first: HF checkpoint dir + GGUF shard path)</code></pre>
<pre><code># 3. Imatrix + quantize with ik_llama.cpp (IQK types are Ik-only)
./build/bin/llama-imatrix -m step5-preview-bf16.gguf \
-f calibration-corpus.txt -o step5.imatrix \
--gpu-layers 12 --ctx-size 8192
./build/bin/llama-quantize --imatrix step5.imatrix \
step5-preview-bf16.gguf step5-preview-IQ4_KS-mix.gguf IQ4_XS</code></pre>
<div class="qz-note">Conversion credit: <a href="https://huggingface.co/avar6/Step5-Preview">avar6</a> (mainline converter patch). The indexer surgery script (<code>indexer_insert.py</code>) is available in this repo's files — edit the CONFIG block at the top with your local paths before running.</div>
<div class="qz-section">Quick Start</div>
<div class="qz-note">Requires <a href="https://github.com/NF09830453/ik_llama.cpp/tree/step5-preview-dense">ik_llama.cpp (Step-5 fork)</a>. Standard llama.cpp will not run these quants.</div>
<span class="qz-code"># Clone and build
git clone --branch step5-preview-dense https://github.com/NF09830453/ik_llama.cpp
cd ik_llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)</span>
<span class="qz-code"># Download (main model)
pip install huggingface_hub
hf download L-Alchemyst/Step-5-Preview-GGUF --repo-type model --include "IQ4_KS-mix/*" --local-dir ./step5-iq4ks</span>
<div class="qz-note">IQ3_KS-mix alternative: swap the include to <code>IQ3_KS-mix/*</code> and point <code>-m</code> at <code>step5-preview-IQ3_KS-mix-00001-of-00030.gguf</code> (30 shards) — same run flags work for both.</div>
<span class="qz-code"># Optional: vision module (mmproj), required for image input
hf download L-Alchemyst/Step-5-Preview-GGUF --repo-type model --include "mmproj-step5-vision-q8_0.gguf" --local-dir ./step5-iq4ks</span>
<div class="qz-note">Hybrid CPU+GPU: adjust <code>-ot</code> to your RAM/VRAM split — <code>-cram</code> is another adjustable flag. The sparse indexer export below is the prod-validated config: <code>IK_SEL_SOFT=0.5</code> is required alongside <code>IK_SPARSE=1</code> (the flag's default 0 is the legacy strict top-k cliff that starves long-context recall). Dense alternative (<code>IK_SPARSE=0</code>) is slightly faster today — sparse attention is mask-only until fused gather kernels land — but sparse matches the imatrix calibration regime.</div>
<span class="qz-code">export IK_SPARSE=1 IK_CSA=2 IK_SSMAX=1 IK_SEL_SOFT=0.5
./build/bin/llama-server \
-m ./step5-iq4ks/step5-preview-IQ4_KS-mix-00001-of-00036.gguf \
--alias Alchemyst/Step-5 \
-ngl 999 \
-ot "blk\.(2[1-9]|[3-8][0-9]|90)\.(ffn_up|ffn_gate|ffn_down)_exps=CPU" \
-ub 4096 -b 4096 \
--flash-attn on \
--ctx-checkpoints 5 \
--swa-compress \
-c 80000 \
-cuda "offload-batch-size=128" \
--spec-type mtp:n_max=1,p_min=0.0 \
-muge \
-gr -ger \
--merge-qkv \
-ctk q8_0 -ctv q8_0 \
-khad -vhad \
--jinja \
-rtr \
--threads 96 --threads-batch 128 \
--mmproj ./step5-iq4ks/mmproj-step5-vision-q8_0.gguf \
--no-mmproj-offload \
--port 8080</span>
<div class="qz-note">The <code>--no-mmproj-offload</code> flag keeps the vision module in system RAM instead of VRAM, trading a little image-processing speed for more GPU headroom.</div>
<div class="qz-note">CPU-only: use <code>-ctk q8_0</code> and prefix with <code>numactl -N ${SOCKET} -m ${SOCKET}</code>.</div>
<div class="qz-credit">
<strong>Credits</strong><br>
Model: <a href="https://huggingface.co/stepfun-ai">stepfun-ai</a><br>
Runtime: <a href="https://github.com/NF09830453/ik_llama.cpp/tree/step5-preview-dense">ik_llama.cpp (Step-5 fork)</a>
</div>
</div>
</body>
</html>
Run L-Alchemyst/Step-5-Preview-GGUF with guIDE
Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.
Source: Hugging Face · Compare models