---

license: apache-2.0

base_model: empero-ai/Qwen3.8-2B-Distill

base_model_relation: quantized

library_name: gguf

pipeline_tag: text-generation

language:

• en

tags:

• gguf
• llama.cpp
• q4_0
• arm
• android
• on-device
• qwen3.5
• qwen3.8

---

Qwen3.8-2B-Distill, pure Q4_0 GGUF for ARM phones

What it is: empero-ai/Qwen3.8-2B-Distill as a single

llama.cpp file in which every weight matrix is Q4_0 (llama-quantize --pure), made from the publisher's BF16 GGUF.

On ARM CPUs with the i8mm or dotprod instructions llama.cpp runs this format faster than the K-quants. On a Snapdragon

phone it reads prompts 1.59x as fast and generates 1.17x as fast as the publisher's recommended

Q4_K_M, and the file is 16 % smaller. The price is a measurable accuracy loss compared with Q4_K_M (details below).

I use it as the on-device chat model of an Android voice assistant (llama.cpp, CPU only).

| | this repo: Q4_0 (pure) | publisher's Q4_K_M |

|---|---:|---:|

| file | Qwen3.8-2B-Q4_0.gguf | Qwen3.8-2B-Q4_K_M.gguf |

| size | 1,105,627,520 B (1.11 GB) | 1,312,164,224 B (1.31 GB) |

| prompt speed, phone CPU (pp256, mean of 2 runs) | 110 t/s | 69 t/s |

| generation speed, phone CPU (tg64, mean of 2 runs) | 17.7 t/s | 15.1 t/s |

| mean KL divergence vs BF16 (lower is better) | 0.086 | 0.041 |

| same most-likely token as BF16 | 84.7 % | 90.1 % |

| perplexity vs BF16 (wikitext-2) | +8.1 % | +3.6 % |

In short, measured on the phone described below: Q4_0 reads prompts 1.59x as fast and generates 1.17x as fast as

Q4_K_M (back-to-back runs), and in exchange it deviates about twice as much from the full-precision model: mean KL

divergence 0.086 vs 0.041, the same most-likely next token as BF16 84.7 % vs 90.1 % of the time, perplexity +8.1 % vs

+3.6 % (llama.cpp's KL-divergence test on wikitext-2).

Base model, credit and licence

• Model: Qwen3.8-2B-Distill by Empero, a

full-parameter distillation into Qwen/Qwen3.5-2B (Alibaba Qwen team).

All credit for the model goes to them; please read the

publisher's model card for what it is good and bad at.

• Source file: Qwen3.8-2B-BF16.gguf from

empero-ai/Qwen3.8-2B-Distill-GGUF at commit

f4f73582d0b149595450c719b9a7521a03894f9c (sha256 44763f3d…ac48, checked).

• Licence: Apache-2.0, as the base model. LICENSE is the licence file shipped with Qwen3.5-2B; NOTICE lists

what was changed. Only the weight format changed: tokenizer, chat template and all other metadata are identical to

the source file (only general.file_type differs). This repo is not affiliated with Empero or Alibaba.

Why Q4_0 is faster on ARM (repacking)

When llama.cpp's CPU backend loads a model it can repack the weights: it rearranges them into an interleaved

layout so that the matrix-multiply instructions of current ARM cores (SMMLA from the i8mm extension, SDOT from dotprod)

process several rows at once. For Q4_0 on an i8mm core it uses 4x8 interleaved blocks. On this phone, the same Q4_0

file reads prompts at 109.9 t/s with repacking and 47.3 t/s with --no-repack (2.3x); generation is the same either

way (17.1 vs 17.0 t/s), because generating one token is a matrix-vector product.

Recent llama.cpp builds (this one included) repack Q4_K and Q6_K too, so the Q4_K_M file is repacked as well, but it

still trails pure Q4_0 here. Q4_0 is the simplest 4-bit block format (32 weights and one scale), while Q4_K_M mixes Q4_K

super-blocks (packed 6-bit sub-scales and minimums) with Q6_K tensors, and it is 19 % more data to read for every

generated token. --pure keeps every matrix in Q4_0, including the token embedding table, which this model also uses as

its output layer.

The cost: the repacked copy lives in ordinary app memory (see Memory), and Q4_0 is clearly less accurate than Q4_K_M

(about twice the KL divergence, see Accuracy).

Speed on the phone

Nothing Phone (4a) Pro, Snapdragon SM7750 (1x Cortex-A720 at 2.8 GHz, 4x A720 at 2.4 GHz, 3x A520 at 1.84 GHz), 12 GB

RAM. llama.cpp b11671 (commit ba3c05f), CPU backend built on the phone with GCC 15.2 (-DGGML_NATIVE=ON, i8mm +

dotprod + bf16 detected) and without OpenMP (-DGGML_OPENMP=OFF). llama-bench, 3 repetitions, threads pinned to the

big cores, flash attention on, --poll 100, --cpu-strict 1.

Back-to-back runs on 2026-10-17, in the order listed (4 big cores, cpu 3-6, -t 4, KV cache F16; phone charging and

warm):

| file | prompt, pp256 (t/s) | generation, tg64 (t/s) | peak RSS (llama-bench) |

|---|---:|---:|---:|

| Q4_0 (this repo), run 1 | 115.7 ± 0.6 | 18.06 ± 0.08 | 2.17 GB |

| publisher's Q4_K_M, run 1 | 69.5 ± 0.4 | 15.10 ± 0.08 | 2.56 GB |

| publisher's Q4_K_M, run 2 | 69.0 ± 1.5 | 15.12 ± 0.16 | 2.56 GB |

| Q4_0 (this repo), run 2 | 104.8 ± 7.6 | 17.35 ± 0.01 | 2.17 GB |

| Q4_0 / Q4_K_M | 1.59x | 1.17x | 0.85x |

An earlier session on a cool, quiet phone gave 127.6 ± 1.6 / 18.76 ± 0.31 t/s for Q4_0 and 76.7 ± 0.3 / 15.00 ± 0.17

t/s for Q4_K_M (1.66x / 1.25x).

Other measurements with this file (same build and phone, earlier sessions):

| setting | prompt t/s | generation t/s |

|---|---:|---:|

| 4 big cores (cpu 3-6), -t 4 | 127.6 ± 1.6 | 18.76 ± 0.31 |

| 5 big cores (3-7, adds the prime core), -t 5 | 147.5 ± 7.1 | 18.81 ± 1.03 |

| 4 big cores + KV cache q8_0 | 130.1 ± 0.4 | 19.55 ± 0.13 |

| 8 threads on all cores (little cores included) | 60.5 ± 10.0 | 13.57 ± 0.49 |

| same commit with OpenMP | 120.1 ± 1.1 | 8.0 ± 0.3 |

| repacking on / off (--no-repack), one session | 109.9 / 47.3 | 17.07 / 16.95 |

| GPU, OpenCL on the Adreno 722, -ngl 99 | 211.5 ± 0.6 | 15.1 ± 0.5 |

In llama-server with the system prompt already cached, a short chat turn starts its answer in about 0.3 s (median

first token 0.28 s, generation median 17.4 t/s over 22 turns).

Accuracy

Next-token distribution vs the full-precision model

llama-perplexity --kl-divergence on the wikitext-2-raw test split (the file llama.cpp's

scripts/get-wikitext-2.sh downloads), first 40 chunks of 512 tokens (10,200 scored tokens). The reference is the

publisher's BF16 weights; for speed they were run as an F16 copy (see the note under the table).

| file | size | mean KLD | 99th pct KLD | same top token | perplexity | PPL vs reference |

|---|---:|---:|---:|---:|---:|---:|

| reference: BF16 weights (as F16) | 3.90 GB | 0 | 0 | 100 % | 10.32 ± 0.27 | 1 |

| publisher's Q4_K_M | 1.31 GB | 0.0410 ± 0.0006 | 0.285 | 90.06 ± 0.30 % | 10.70 ± 0.29 | 1.0359 ± 0.0032 |

| Q4_0 (this repo) | 1.11 GB | 0.0856 ± 0.0012 | 0.554 | 84.72 ± 0.36 % | 11.16 ± 0.30 | 1.0810 ± 0.0048 |

(Mean KLD: the average KL divergence between the reference's and the quantized model's next-token distributions; 0

means identical. "Same top token": how often both pick the same most likely next token. Uncertainties are llama.cpp's

standard errors.)

Reading: pure Q4_0 moves this model about twice as far from the original as Q4_K_M does. Its perplexity is 8.1 %

above the reference (Q4_K_M: 3.6 %), and its most likely next token differs from the original's on 15.3 % of positions

(Q4_K_M: 9.9 %). Most of those differences are between near-equal candidates (median KLD 0.060, against 0.026 for

Q4_K_M), but the tail is heavier too (99th percentile 0.55 vs 0.29). This is the accuracy you give up for the speed.

Reference note: llama.cpp has no fast BF16 matrix path on ARM CPUs (BF16 took 202 s per chunk here, F16 22 s), so the

BF16 weights were converted to F16 with llama-quantize … F16. F16 holds every weight of this model exactly except the

0.47 % with magnitude below 2^-14, which become F16 subnormals (absolute error below 3e-8); the largest weight is 0.91,

far from the F16 limit. Check on the first 2 chunks (510 scored tokens), with the original BF16 weights on the slow

path as the reference: the F16 copy has mean KLD 0.00004 (max 0.0009, same top token 99.4 %), while this Q4_0 has

0.081 on the same tokens. The F16 stand-in is about 2,000x closer to BF16 than the quantization being measured.

Assistant answers

This file is the one I tested in a private 68-prompt eval of the voice-assistant use case: 40 everyday prompts and 28

harder maths, reasoning and instruction-following prompts with written reference answers, thinking off, the chat

settings below, one sample per prompt. Automatic answer checks (the expected answer appears, or the exact requested

format is kept) passed on 14 of 16 checkable everyday prompts and 13 of 20 checkable hard prompts.

A Q4_0 vs Q4_K_M comparison on this eval was not run, so for this model the accuracy cost of Q4_0 is known only

from the KL-divergence numbers above. (With 20 checkable prompts and sampling at temperature 0.3, a difference of one or

two answers would be within run-to-run noise anyway.)

The publisher's own benchmarks (context)

Empero reports, for the BF16 model with chain-of-thought: MMLU (CoT) 0.548 and GSM8K (CoT) 0.640, against 0.283 and

0.330 for the Qwen3.5-2B base. These are the publisher's numbers, not re-measured here, and not for this quantization.

Memory

Repacking puts a rearranged copy of the weights in anonymous (non-reclaimable) memory; the memory-mapped GGUF pages

are only page cache that the OS can drop. Measured in llama-server right after load: **RSS 2.23 GB = 1.20 GB anonymous

(repacked weights and buffers) + 1.03 GB file pages; 2.31 GB after a few turns. Plan for about 2.3-2.6 GB free RAM**.

With --no-repack the weights stay file-backed (the OS can evict them) but prompt speed falls to about 47 t/s.

How it was made

# source: empero-ai/Qwen3.8-2B-Distill-GGUF @ f4f73582d0b149595450c719b9a7521a03894f9c, Qwen3.8-2B-BF16.gguf
llama-quantize --pure Qwen3.8-2B-BF16.gguf Qwen3.8-2B-Q4_0.gguf Q4_0 4

The published file was made on 2026-10-05 with llama-quantize from llama.cpp commit 11fe021

(11fe02151f79c41d0d4af7da708755d73b9c0da6). All measurements on this card used b11671 (ba3c05f). Re-running the same

command with b11671 gives a file of exactly the same size (1,105,627,520 B) but not byte-identical, so the checksum

below is specific to the 11fe021 build.

Accuracy commands (see eval/ for the full logs):

llama-quantize Qwen3.8-2B-BF16.gguf Qwen3.8-2B-F16.gguf F16 4          # reference copy (see note above)
llama-perplexity -m Qwen3.8-2B-F16.gguf -f wikitext-2-raw/wiki.test.raw -c 512 -b 512 --chunks 40 -t 4 \
  --kl-divergence-base qwen-f16.kld
llama-perplexity -m Qwen3.8-2B-Q4_0.gguf   --kl-divergence-base qwen-f16.kld --kl-divergence -c 512 -b 512 -t 4
llama-perplexity -m Qwen3.8-2B-Q4_K_M.gguf --kl-divergence-base qwen-f16.kld --kl-divergence -c 512 -b 512 -t 4

Recommended llama.cpp settings (ARM phone)

llama-server -m Qwen3.8-2B-Q4_0.gguf --jinja -c 4096 \
  -t 4 -C 0x78 --cpu-strict 1 --poll 100 -fa on -ctk q8_0 -ctv q8_0

• Threads on the big cores only. -C 0x78 pins to cpus 3-6, the four 2.4 GHz A720 cores of the SM7750; use the mask

of your phone's big cores. Adding the prime core (-C 0xF8 -t 5) gives +15 % prompt speed but no generation speed and

it throttles first. Never include the little cores: 8 threads cost 28 % generation and 53 % prompt speed.

• Build without OpenMP (-DGGML_OPENMP=OFF): ggml's own thread pool with polling generated 18.8 t/s vs 8.0 t/s with

OpenMP on the same commit.

• Keep repacking on (the default). -fa on; KV cache q8_0 costs nothing here (only one layer in four is attention).
• --poll 50 or more is about 4 % faster than --poll 0.
• Cache the system prompt (llama-server slot save/restore) so a new chat starts in ~0.3 s instead of ~1.5 s.

Chat settings that worked for a short-answer voice assistant

• Thinking off: chat_template_kwargs: {"enable_thinking": false}. This is a reasoning model; with thinking on, a

simple greeting takes several seconds before the answer starts.

• Sampler: temperature 0.3, top_p 0.95, top_k 20, max_tokens about 512. (The publisher recommends 0.6 for normal,

thinking use; 0.3 fixed two clock-arithmetic slips in my tuning set and was the fastest.)

• System prompt: short, voice-style: answer in one to three plain sentences, no markdown, admit when a live fact

(date, weather, news) is unknown; two one-line example exchanges and "work arithmetic out step by step" helped.

• The model still writes lists and bold text for some requests, so a text-to-speech app should strip markdown.

Limitations

• Less accurate than Q4_K_M: see the tables. If you have the RAM and speed headroom, Q4_K_M or Q5/Q6 files from the

publisher are closer to the original.

• Speeds are for one phone and one llama.cpp build. x86 CPUs also repack Q4_0 (AVX2) but were not measured.
• Repacking roughly doubles resident memory while the file pages are still cached.
• Accuracy here was measured on English wikitext and a small private English eval only.
• All limitations of the base model apply (a 2B model: limited factual recall, it can state wrong facts confidently,

and with thinking off it is weaker at multi-step maths).

Files and checksums

| file | bytes | sha256 |

|---|---:|---|

| Qwen3.8-2B-Q4_0.gguf | 1,105,627,520 | 2662c3bd5e028e7eb73b21902d0a8198370c6f426bd5944dee2b714cf8269089 |

| LICENSE | | Apache-2.0 (from Qwen/Qwen3.5-2B) |

| NOTICE | | attribution and changes |

| eval/ | | KL-divergence logs, speed numbers and results.json |