---

license: llama3.1

license_name: llama3.1

license_link: https://www.llama.com/llama3_1/license/

base_model: meta-llama/Llama-3.1-8B-Instruct

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- llama

- llama-3.1

- gguf

- q4_k_m

- calibration-aware-quantization

- fraqtl

---

Llama-3.1-8B-Instruct Q4_K_M GGUF — fraQtl Hi-Fi (calibration-aware quantization)

Built with Llama.

Iso-size Q4_K_M-class GGUF of meta-llama/Llama-3.1-8B-Instruct built with

fraQtl calibration-aware per-tensor quantization: same file-size class as the

leading community Q4_K_M (within 0.007%), measurably closer to the original

bf16 model's output distribution on both locked evaluation slices.

This is the target half of the fraQtl Llama Draft Pair (its

speculative-decoding draft is

fraQtl/Llama-3.2-1B-Instruct-Hi-Fi-GGUF);

the pair receipt ships in the collection when its benches complete.

Evaluation — fidelity to the original model

Metric: symmetric top-20 KLD on the support of a Q8_0 teacher **quantized from

the original pinned bf16 checkpoint** (this measures fidelity-to-original — the

axis a user migrating from the bf16 model experiences). 3 independent runs,

run-to-run drift 0.0 (deterministic rig). Baseline credited:

bartowski

Q4_K_M — the canonical community quantization (pinned revision bf5b95e9…).

| Arm | bytes | true bpw | KLD code/math ↓ | KLD general ↓ | top-1 agree cm | top-1 agree gen |

|---|---|---|---|---|---|---|

| bartowski Q4_K_M | 4,920,739,232 | 4.9022 | 0.034463 | 0.022621 | 0.9532 | 0.9470 |

| fraQtl Hi-Fi (this repo) | 4,920,411,392 (−0.007%) | 4.9019 | 0.028341 (−17.8%) | 0.019196 (−15.1%) | 0.9580 | 0.9510 |

Slices: code/math 400-record held-out (sha cce68602…b1665) and general

chat/tool/wikitext 400-record (sha b10a79ca…276). Eval runtime:

llama-cpp-python==0.3.33 (vendored llama.cpp 78d2f5246), A100-80GB, seed 0.

Long-context retrieval (needle-in-a-haystack)

Grid: contexts {8192, 32768} × depths {0.1, 0.5, 0.9} × 3 passkeys (18 cells),

exact-match, greedy, 3 runs per arm. **Both this artifact and the bartowski

baseline score 18/18 on every run** — the grid saturates at 8B scale; we state

parity, not a win. Note: greedy decoding, but CUDA reduction order varies

across instances, so retrieval scores are always reported as multi-run.

Provenance

| Field | Value |

|---|---|

| Base model | meta-llama/Llama-3.1-8B-Instruct @ 0e9e39f249a16976918f6564b8830bc894c89659 |

| Conversion + quantization | llama.cpp @ 78d2f524682d9fee790a6460c93d018dafeb5229; convert_hf_to_gguf.py --outtype bf16, then llama-quantize with per-tensor --tensor-type overrides |

| Calibration | fraQtl code+math packet (sha256 9bec84a2…29e02), llama-imatrix ctx 2048, BF16 substrate; imatrix.dat included in this repo |

| Teacher | Q8_0 quantized from the original bf16 at the same llama.cpp commit |

| Artifact sha256 | 3c9656099140a679fe3a4061a61498dcb89fc324d978769312f4311a5de642da |

| Receipts | receipts/ in this repo: build + evaluation + retrieval JSONs, per-layer true-bpw table |

Architecture / allocation

Standard dense Llama (32 layers, GQA). The type placement is chosen per tensor

by calibration-aware allocation at the identical total byte budget as the

baseline — the claim is smarter bits, not more bits. Attention V is the most

protected family (Q8_0 on 29/32 layers); attention Q is never demoted below the

baseline's native type. The lm_head stays at the baseline's Q6_K.

Usage

# llama.cpp
llama-cli -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "your prompt" --temp 0.7

# speculative decoding with the fraQtl 1B draft (pair receipt to follow)
llama-speculative -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  -md Llama-3.2-1B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  --draft-max 16 -ngl 99 -ngld 99 -p "your prompt"

The GGUF embeds the chat template; use llama_chat_apply_template /

chat-completion APIs.

fraQtl ladder

• Hi-Fi GGUFs (this repo + the 1B draft) — download-and-run quality tier
• Qwen3-4B KV sidecars — the serving lane (long-context KV compression receipts)

License

Llama 3.1 Community License (full text in LICENSE; attribution in NOTICE).

Derivative model name begins with "Llama" per the license. Use is subject to

Meta's Acceptable Use Policy referenced therein.

Limitations

• Fidelity numbers measure distributional closeness to the original model, not

task accuracy; no task-benchmark claims are made on this card.

• Q4_K_M-class quantization: outputs can differ from the bf16 model.
• The retrieval grid saturates for both arms at 8B — it differentiates the 1B

tier, not this one.

The fraQtl ladder

| Tier | Job | Example |

|---|---|---|

| Hi-Fi | maximum fidelity at standard 4-bit size | this repo's Q4_K_M |

| Hi-Fi Phone | fits phone/laptop RAM, fidelity-first | E2B Phone, 2.86 GB |

| SmartEdge | smallest footprint that survives | E2B IQ3XXS, 2.45 GB |

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.