---

license: other

license_name: llama4-community-license

license_link: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

base_model: meta-llama/Llama-4-Scout-17B-16E-Instruct

base_model_relation: quantized

pipeline_tag: text-generation

library_name: gguf

tags:

- gguf

- moe

- apex

- quantized

- llama.cpp

- llama4

---

Llama 4 Scout 17B-16E Instruct — APEX quants

Measured/structural per-tensor bit allocation (APEX) GGUF quants of

meta-llama/Llama-4-Scout-17B-16E-Instruct,

built entirely from a full-precision BF16 GGUF (no safetensors ever downloaded — see Method below).

⚠️ Set --ctx-size explicitly — do not run this model with defaults

Llama 4 Scout was trained with up to a 10,485,760-token (10M) context window. If you launch

llama-cli / llama-server without an explicit --ctx-size, llama.cpp defaults the KV cache

to the model's own trained context length — not a small sane default. For most models that's just

an oversized-but-harmless KV cache; for this model it means **an attempt to allocate a KV cache

sized for 10 million tokens**, which can consume many terabytes of memory and hard-lock a machine

(this happened to us during testing — full physical reboot required, not a clean OOM kill).

Always pass --ctx-size sized to what you actually need and what your hardware can hold, e.g.:

llama-server -m Llama-4-Scout-17B-16E-Instruct-APEX-i-compact.gguf --ctx-size 8192 ...
llama-cli    -m Llama-4-Scout-17B-16E-Instruct-APEX-i-compact.gguf --ctx-size 8192 -p "..." -st

8K–32K is plenty for most chat/tool-use workloads. Only reach for six- or seven-figure context

sizes if you have the RAM/VRAM to back it, and scale up deliberately — never rely on the default.

Sizes

FYI, these models are big and I'm on Starlink. I'll upload models as I can.

| size | quant mix | file size | bits/weight |

|---|---|---|---|

| fp (BF16, reference) | — | ~205.6 GiB | 16.00 |

| i-quality | Q6_K attn/ffn, Q8_0 shared-expert, Q4_K routed-expert baseline | 71,857.97 MiB (≈75.4 GB) | 5.59 |

| i-compact | Q6_K attn, Q8_0 shared-expert, Q4_K routed-expert | 52,297.97 MiB (≈54.9 GB) | 4.07 |

| i-mini | Q6_K attn, Q8_0 shared-expert, IQ3_XXS routed-expert | 36,054.85 MiB (≈37.8 GB) | 2.81 |

token_embd.weight and output.weight are held at Q6_K in all three tiers.

Validation status: allocation is structural (role/depth-band priors), not measured —

Scout is too large for this pipeline's current measured-sensitivity mode (which needs the whole

model resident in memory for per-tensor forward passes; a memory-bounded version is future work).

Real-generation coherence and tool-calling checks are pending a re-run and not yet reflected

in this card — an in-progress gate run for i-mini was interrupted by the context-size incident

described above, before the fix landed. Treat these quants as built-and-untested until that gate

run completes; this README will be updated with results.

No wikitext PPL-ratio baseline: the fp BF16 model (~205 GiB) doesn't fit resident on the build

hardware (119 GB unified memory) to run a reference perplexity pass.

Method

Built without ever downloading Scout's ~217 GB of raw safetensors. The importance matrix (imatrix)

was computed by a band-serialized generator that reads weights directly from a pre-converted

full-precision BF16 GGUF (unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF), materializing one

transformer layer at a time — same approach validated byte-exact against the safetensors path on

a smaller model first (granite-3.1) before trusting it on Scout. This keeps peak memory bounded

regardless of total model size, which is what made a 109B-parameter (17B active) MoE tractable on

a single machine with 119 GB of unified memory.

Bit allocation is structural: attention and shared-expert tensors get higher precision than routed

expert tensors, following role/depth-band priors rather than per-tensor measured sensitivity.

Calibration

Imatrix computed over Salesforce/wikitext,

126 chunks × 512 tokens — the same calibration corpus used across this project's other APEX quants.

Attribution & licenses

• Base: meta-llama/Llama-4-Scout-17B-16E-Instruct (Llama 4 Community License — see base model repo for terms)
• Engine: llama.cpp (MIT)
• APEX: localai-org/apex-quant (@mudler) (MIT)
• Calibration: Salesforce/wikitext

Unofficial community quantization; not affiliated with or endorsed by Meta or the base model's

publisher. Use of this model is subject to the Llama 4 Community License Agreement linked above.