---

license: llama3.2

license_name: llama3.2

license_link: https://www.llama.com/llama3_2/license/

base_model: meta-llama/Llama-3.2-1B-Instruct

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- llama

- llama-3.2

- gguf

- q4_k_m

- calibration-aware-quantization

- speculative-decoding-draft

- fraqtl

---

Llama-3.2-1B-Instruct Q4_K_M GGUF — fraQtl Hi-Fi (calibration-aware quantization)

Built with Llama.

Iso-size Q4_K_M-class GGUF of meta-llama/Llama-3.2-1B-Instruct built with

fraQtl calibration-aware per-tensor quantization. Same file-size class as the

leading community Q4_K_M (within 0.02%). Wins the general slice by a wide

margin and top-1 agreement on both slices; **on the code/math KLD slice it is

0.11% behind the baseline** — stated plainly below, both directions.

This is the draft half of the fraQtl Llama Draft Pair (its target is

fraQtl/Llama-3.1-8B-Instruct-Hi-Fi-GGUF);

the speculative-decoding pair receipt ships in the collection when its benches

complete.

Evaluation — fidelity to the original model

Metric: symmetric top-20 KLD on the support of a Q8_0 teacher **quantized from

the original pinned bf16 checkpoint** (fidelity-to-original — the axis a user

migrating from the bf16 model experiences). 3 independent runs, run-to-run

drift 0.0 (deterministic rig). Baseline credited:

bartowski

Q4_K_M — the canonical community quantization (pinned revision 067b946c…).

| Arm | bytes | true bpw | KLD code/math ↓ | KLD general ↓ | top-1 agree cm | top-1 agree gen |

|---|---|---|---|---|---|---|

| bartowski Q4_K_M | 807,694,464 | 5.2286 | 0.046045 | 0.036032 | 0.9369 | 0.9227 |

| fraQtl Hi-Fi (this repo) | 807,513,952 (−0.02%) | 5.2274 | 0.046093 (+0.11% — behind baseline) | 0.031523 (−12.5%) | 0.9416 | 0.9297 |

Read the table exactly as printed: this artifact loses the code/math KLD

comparison by 0.11% relative and wins the general KLD comparison by 12.5%

relative; it wins top-1 agreement on both slices. Slices: code/math 400-record

held-out (sha cce68602…b1665), general chat/tool/wikitext 400-record (sha

b10a79ca…276). Eval runtime: llama-cpp-python==0.3.33 (vendored llama.cpp

78d2f5246), A100-80GB, seed 0.

Long-context retrieval (needle-in-a-haystack)

Grid: contexts {8192, 32768} × depths {0.1, 0.5, 0.9} × 3 passkeys (18 cells),

exact-match, greedy. Greedy decoding still shows ±1-cell flicker across

container instances (CUDA reduction order), so scores are multi-run ranges,

never single-run points:

| Arm | 18-cell score (4 runs ours / 3 runs baseline) | 32K subgrid |

|---|---|---|

| fraQtl Hi-Fi | 17–18 / 18 | 9/9 every run |

| bartowski Q4_K_M | 16–17 / 18 | 9/9 every run |

This artifact never scored below the baseline in any paired run.

Provenance

| Field | Value |

|---|---|

| Base model | meta-llama/Llama-3.2-1B-Instruct @ 9213176726f574b556790deb65791e0c5aa438b6 |

| Conversion + quantization | llama.cpp @ 78d2f524682d9fee790a6460c93d018dafeb5229; convert_hf_to_gguf.py --outtype bf16, then llama-quantize with per-tensor --tensor-type overrides |

| Calibration | fraQtl mixed code+math / general packet (sha256 c88e7e0a…2dabc), llama-imatrix ctx 2048, BF16 substrate; imatrix.dat included |

| Teacher | Q8_0 quantized from the original bf16 at the same llama.cpp commit |

| Artifact sha256 | 241b1e4848762cc77f2ebee9090e6a898f8541f421f7d854509f5de3a51452bb |

| Receipts | receipts/ in this repo: build + evaluation + retrieval JSONs, per-layer true-bpw table |

Architecture / allocation

Standard dense Llama (16 layers, GQA, tied embeddings). Type placement chosen

per tensor by calibration-aware allocation at the identical total byte budget

as the baseline — smarter bits, not more bits. Attention V is the most

protected family (Q8_0 on 12/16 layers); attention Q is never demoted below

the baseline's native type; the tied token-embedding/output table stays at the

baseline's Q6_K.

Usage

# standalone
llama-cli -m Llama-3.2-1B-Instruct-fraQtl-HiFi-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "your prompt" --temp 0.7

# as the speculative-decoding draft for the fraQtl 8B (pair receipt to follow)
llama-speculative -m Llama-3.1-8B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  -md Llama-3.2-1B-Instruct-fraQtl-HiFi-Q4_K_M.gguf \
  --draft-max 16 -ngl 99 -ngld 99 -p "your prompt"

The GGUF embeds the chat template; use llama_chat_apply_template /

chat-completion APIs.

fraQtl ladder

• Hi-Fi GGUFs (this repo + the 8B target) — download-and-run quality tier
• Qwen3-4B KV sidecars — the serving lane (long-context KV compression receipts)

License

Llama 3.2 Community License (full text in LICENSE.txt, included verbatim;

attribution in NOTICE). Derivative model name begins with "Llama" per the

license. Use is subject to Meta's Acceptable Use Policy referenced therein.

Note: the pinned license text distributed with the text-only 1B model contains

no European-Union-specific restriction clause; Meta's EU-restriction language

in the Llama 3.2 family applies to the multimodal models. The license file in

this repo is the unmodified text distributed with the pinned base revision.

Limitations

• Fidelity numbers measure distributional closeness to the original model, not

task accuracy; no task-benchmark claims are made on this card.

• The code/math KLD slice is 0.11% behind the credited baseline (see table).
• Q4_K_M-class quantization of a 1B model: expect the quality limits of the 1B

tier; this artifact passed all greedy coherence smokes in our gates (receipts

in-repo).

The fraQtl ladder

| Tier | Job | Example |

|---|---|---|

| Hi-Fi | maximum fidelity at standard 4-bit size | this repo's Q4_K_M |

| Hi-Fi Phone | fits phone/laptop RAM, fidelity-first | E2B Phone, 2.86 GB |

| SmartEdge | smallest footprint that survives | E2B IQ3XXS, 2.45 GB |

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.