---

license: apache-2.0

base_model: Qwen/Qwen3-4B-Instruct-2507

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

tags:

- qwen

- qwen3

- gguf

- q4_k_m

- q4_0

- calibration-aware-quantization

- fraqtl

---

Qwen3-4B-Instruct-2507 GGUF — fraQtl Hi-Fi (calibration-aware quantization)

Iso-size Q4_K_M-class GGUF of Qwen/Qwen3-4B-Instruct-2507 built with fraQtl

calibration-aware per-tensor quantization: byte-matched to the leading

community Q4_K_M (within 0.016%), measurably closer to the original model's

output distribution on both locked evaluation slices — plus a **Q4_0

edition** (the NPU-native fixed format) with the calibration lever measured

in isolation.

Evaluation — fidelity to the original model

Metric: symmetric top-20 KLD on the support of a Q8_0 teacher **quantized

from the original pinned bf16 checkpoint** (fidelity-to-original; teacher-

framing caveat applies). All arms evaluated with the identical binary

(llama.cpp 4df29be4), identical rendered prompt template, 3 runs/arm,

drift 0. Baselines credited:

MaziyarPanahi

Q4_K_M (canonical by downloads, ~200k/30d) and

unsloth Q4_K_M.

| Arm | Bytes | KLD code/math ↓ | KLD general ↓ | top-1 cm/gen |

|---|---:|---:|---:|---:|

| fraQtl Hi-Fi (this repo) | 2,496,879,712 (−0.016%) | 0.025317 (−55.9%) | 0.033753 (−46.4%) | 0.9604 / 0.9357 |

| MaziyarPanahi Q4_K_M (canonical) | 2,497,280,448 | 0.057372 | 0.063008 | 0.9466 / 0.9186 |

| unsloth Q4_K_M (second) | 2,497,281,120 | 0.030018 | 0.042626 | 0.9557 / 0.9276 |

Relative deltas vs canonical. The second arm (unsloth) is itself strong — we

beat it too on both slices (−15.7% code/math, −20.8% general), stated for

completeness.

Long-context retrieval — including the full 262K native context

Needle exact-match, 3 depths × 3 keys per context, greedy, multi-run:

| Arm | 8K + 32K (×3 runs) | 262,144-token native context (×2 runs) |

|---|---|---|

| fraQtl Hi-Fi | 18/18, 18/18, 18/18 | 9/9, 9/9 |

| MaziyarPanahi Q4_K_M | 18/18 ×3 (parity) | 9/9 ×2 (parity) |

| unsloth Q4_K_M | 18/18 ×3 (parity) | 8/9 ×2 |

The 262K row runs the model's full native context window with per-cell

receipts in-repo. 8K/32K saturate for all arms — reported as parity, not a

differentiator.

Q4_0 edition — the NPU-native format, calibration lever isolated

Q4_0 is the fixed-format quantization used as the universal GGUF asset by

on-device runtimes. Three arms, same teacher and slices, 3 runs, drift 0:

| Arm | Bytes | KLD code/math ↓ | KLD general ↓ |

|---|---:|---:|---:|

| fraQtl Q4_0 (this repo, our imatrix) | 2,375,771,232 | 0.059665 | 0.080458 |

| naive Q4_0 (no imatrix, same source) | 2,369,545,024 | 0.070135 | 0.088598 |

| GenieX-designated Q4_0 (unsloth's build) | 2,375,773,280 | 0.060996 | 0.077572 |

• Calibration lever, same format: −14.9% code/math and −9.2% general

from calibration alone (ours vs naive, identical Q4_0, identical source).

• The GenieX-designated arm is the community GGUF that

qualcomm/Qwen3-4B-Instruct-2507's release_assets.json (v0.59.0)

designates as the geniex_llamacpp universal asset — built by

unsloth, credited: it is well-made. A

published accuracy datapoint for that artifact: we are −2.2% on code/math

and +3.7% behind on general — both directions stated.

• Provenance note: Qualcomm publishes no GGUF of its own for this model —

its official builds are chipset-specific w4a16 QNN packages (QAIRT 2.45);

the GGUF path is delegated to the community asset above.

• K-quant-on-Hexagon NPU acceleration is unverified by us (no device in

hand); the Q4_0 artifact is provided as the NPU-native-format option.

Disclosures

• Wikitext-2 PPL (ctx 512, sanity metric): ours 10.478, canonical 10.565,

unsloth 10.616 — values stated; ctx-512 chunked raw text on an

instruction-tuned model is a sanity check, not a capability measure.

• The three arms embed different chat templates; all arms were evaluated

with the same rendered template, and each arm's native template sha is

recorded in the build receipt.

• No task-benchmark rows on this card: n=200-class task runs cannot

statistically separate arms of this quality (see our Qwen3.8-27B card's

per-item audit); KLD and retrieval carry the claim.

Runnable receipts — receipts_pack/

Every number above can be re-run with one file on stock llama.cpp — no

custom runtime: receipts_pack/ contains the runner

(run_receipts.py), slice definitions, the needle-grid spec, and the two

teacher logit dumps needed for third-party KLD reproduction.

Provenance

| Field | Value |

|---|---|

| Base model | Qwen/Qwen3-4B-Instruct-2507 @ cdbee75f17c01a7cc42f958dc650907174af0554 (Apache-2.0) |

| Hi-Fi Q4_K_M | sha256 a54cce471d7824219bf13ca4c227b3408bde8591788a28de314933b7a0cb9529 |

| Hi-Fi Q4_0 | sha256 0f7d0956111e721fdde684872ca44b355dc70cb219ae8cf0e059871278ac8888 |

| imatrix | sha256 21b3655ecad1f09f2bd841bcbb48c9ac00693a0d48e296e1fb5263103d4a6b3a (included) |

| Toolchain | llama.cpp @ 4df29be4f4c3673f428170fda944a5b19f743bb8 — conversion, quantization, and every eval |

| Baselines pinned | MaziyarPanahi 953ba5b5…; unsloth 3605803b…; GenieX-designated Q4_0 e0ba675d… |

| Eval runtime | llama-cpp-python 0.3.35, deterministic rig, per-run identity gates |

Receipts for every number: receipts/ (7 JSONs: pins,

conversion gate, calibration selection, frozen recipe, build + 3-run KLD +

PPL + smokes, NIAH grid, Q4_0 arms).

The fraQtl ladder

| Tier | Job | Example |

|---|---|---|

| Hi-Fi | maximum fidelity at standard 4-bit size | this repo's Q4_K_M |

| Hi-Fi Phone | fits phone/laptop RAM, fidelity-first | E2B Phone, 2.86 GB |

| SmartEdge | smallest footprint that survives | E2B IQ3XXS, 2.45 GB |

Same discipline at every tier: pinned provenance, measured numbers, losses

disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds **nine

concurrent ≈128K-context users on a single A100** (134.1 tok/s aggregate,

9/9 per-user retrieval checks, receipt 2026-08-14), on this same base

model: fraQtl/qwen3-4b-instruct-2507-kv-sidecars.

Org page: huggingface.co/fraQtl.