---

license: apache-2.0

base_model:

• Qwen/Qwen3.5-0.8B

tags:

• qwen
• qwen3_5
• imatrix
• conversational
• mtp
• llama.cpp

pipeline_tag: image-text-to-text

---

niki-allocator: Qwen3.5-0.8B quantized models

These quants were produced by an experimental tool, niki-allocator.

The tool is under active development and testing, and has not been published yet.

If you use these quants, I strongly recommend:

• Using the temperature and sampling settings recommended by Alibaba for Qwen3.5-0.8B
• Using this chat template: https://huggingface.co/Moore2877/Qwen-Fixed-Chat-Templates-llamacpp

Qwen3.5-0.8B quantization quality comparison

Per-tensor type allocation vs Unsloth Dynamic 2.0 quants, evaluated with

per-chunk KL divergence against the BF16 model.

Quality vs size

Mean KLD as a function of model size shows the allocator curve consistently

below the unsloth curve, with both corpora overlapping closely:

!Mean KLD by model size, kilo-small

!Mean KLD by model size, wikitext-2

Models

• Model: Qwen3.5-0.8B (MTP/NextN,

25 layers: 24 main + 1 NextN head).

• Baseline: BF16 GGUF (1485.50 MiB), perplexity 6.055 on kilo-small corpus,

15.508 on wikitext-2 test.

• Per-tensor allocator quants (8 sizes): generated by a prototype per-tensor

type allocator — each tensor assigned independently to the quantization type

(Q5_K, Q6_K, IQ4_XS, etc.) that minimizes its proxy reconstruction error

under a hard target size. Quantization performed by llama.cpp's own

quantizer; imatrix-guided scale optimization used where the quantizer

supports it. Source GGUF, imatrix, and assignment logs preserved.

• unsloth quants (8 sizes): unsloth/Qwen3.5-0.8B-MTP-GGUF,

produced with Unsloth Dynamic 2.0 quantization (7 of 8 use the UD_ prefix:

UD_Q2_K_XL, UD_Q3_K_XL, UD_Q4_K_XL, UD_Q5_K_XL, UD_Q6_K_XL, UD_IQ2_M,

UD_IQ3_XXS; IQ4_NL has no prefix and is a more conventional quant), same

model, MTP preserved.

Sizes were chosen to match: each allocator quant targets the size of an

unsloth quant (e.g. allocator 511 MiB vs unsloth IQ4_NL 510.59 MiB).

| Allocator size | Unsloth quant | Unsloth size |

|---|---|---|

| 390.00 MiB | UD_IQ2_M | 390.37 MiB |

| 409.00 MiB | UD_IQ3_XXS | 409.23 MiB |

| 434.00 MiB | UD_Q2_K_XL | 433.52 MiB |

| 484.99 MiB | UD_Q3_K_XL | 484.54 MiB |

| 510.99 MiB | IQ4_NL | 510.59 MiB |

| 545.99 MiB | UD_Q4_K_XL | 546.44 MiB |

| 585.99 MiB | UD_Q5_K_XL | 585.75 MiB |

| 753.97 MiB | UD_Q6_K_XL | 754.21 MiB |

Allocator type distributions vary by size: smaller targets use cheaper types

(IQ4_XS, IQ3_S, Q3_K), larger targets transition to Q5_K/Q6_K/Q8_0. MTP/NextN

tensors received types equal to or better than the unsloth quant's base type at the

same size; quality gains are not achieved by sacrificing MTP heads.

Evaluation methodology

Metric: KL divergence per evaluation chunk between the quantized model's

token probabilities and the BF16 baseline, computed by llama-perplexity with

--saved-logits (full probability vectors saved, compared offline). Reported

metrics: mean KLD, p95/p99/p99.9 KLD, maximum KLD (worst chunk), RMS Δp,

same-top probability, top-flip weight, mean perplexity of the quant.

Corpora:

• kilo-small: ~290K tokens, hold-out from the calibration corpus (agent

session transcripts: English, Russian, code). Used for the primary comparison

(all 16 models evaluated).

• wikitext-2 test: ~290K tokens, standard test sample. Used for a subset

of models to check generalization outside the calibration domain.

Chunk size: 2304 tokens per chunk (not the default 512). Reasoning models

benefit from longer context windows during evaluation; 2304 was the fastest

working size on the evaluation hardware. Reported KLD values are per-chunk

averages and are not directly comparable to evaluations using 512-token chunks.

KLD bases: BF16 logits saved once per corpus; all quants compared against

the same base.

Results: kilo-small corpus (all models)

| Model | Size (MiB) | Mean PPL | Mean KLD | p95 KLD | p99 KLD* | p99.9 KLD* | Max KLD | RMS Δp | Same top p | Top flip weight |

|---|---|---|---|---|---|---|---|---|---|---|

| BF16 baseline | 1485.50 | 6.055 | — | — | — | — | — | — | — | — |

| allocator 390 | 390.00 | 6.808 | 0.1194 | 0.3741 | 0.6519 | 1.3625 | 11.1072 | 9.254 | 84.255 | 0.021070 |

| unsloth UD_IQ2_M | 390.37 | 8.718 | 0.3783 | 1.1583 | 1.9703 | 3.7493 | 10.0537 | 17.182 | 73.733 | 0.062213 |

| allocator 409 | 409.00 | 6.590 | 0.0896 | 0.2763 | 0.4807 | 1.0326 | 6.7363 | 8.163 | 85.817 | 0.016525 |

| unsloth UD_IQ3_XXS | 409.23 | 7.617 | 0.2411 | 0.7498 | 1.3027 | 2.5299 | 7.8096 | 13.354 | 78.648 | 0.040041 |

| allocator 434 | 434.00 | 6.716 | 0.1051 | 0.3344 | 0.5902 | 1.2397 | 8.3567 | 8.639 | 85.431 | 0.018462 |

| unsloth UD_Q2_K_XL | 433.52 | 8.227 | 0.3261 | 1.0307 | 1.7293 | 3.2185 | 9.6882 | 15.395 | 75.580 | 0.053019 |

| allocator 485 | 484.99 | 6.282 | 0.0391 | 0.1223 | 0.2104 | 0.4446 | 2.4922 | 5.276 | 90.614 | 0.007151 |

| unsloth UD_Q3_K_XL | 484.54 | 6.575 | 0.0857 | 0.2693 | 0.4706 | 0.9816 | 4.5137 | 7.883 | 86.402 | 0.015512 |

| allocator 511 | 510.99 | 6.209 | 0.0267 | 0.0839 | 0.1472 | 0.3306 | 1.9196 | 4.345 | 92.246 | 0.004852 |

| unsloth IQ4_NL | 510.59 | 6.313 | 0.0453 | 0.1422 | 0.2458 | 0.5262 | 2.8035 | 5.607 | 89.983 | 0.007999 |

| allocator 546 | 545.99 | 6.194 | 0.0247 | 0.0786 | 0.1405 | 0.3189 | 1.6694 | 4.119 | 92.640 | 0.004316 |

| unsloth UD_Q4_K_XL | 546.44 | 6.248 | 0.0300 | 0.0956 | 0.1630 | 0.3332 | 1.9140 | 4.545 | 91.760 | 0.005340 |

| allocator 586 | 585.99 | 6.106 | 0.0109 | 0.0343 | 0.0596 | 0.1363 | 0.6685 | 2.785 | 94.789 | 0.001985 |

| unsloth UD_Q5_K_XL | 585.75 | 6.128 | 0.0119 | 0.0372 | 0.0624 | 0.1221 | 0.7350 | 2.833 | 94.573 | 0.002085 |

| allocator 754 | 753.97 | 6.073 | 0.0023 | 0.0072 | 0.0123 | 0.0261 | 0.1577 | 1.288 | 97.322 | 0.000368 |

| unsloth UD_Q6_K_XL | 754.21 | 6.072 | 0.0027 | 0.0084 | 0.0140 | 0.0284 | 0.1662 | 1.388 | 97.088 | 0.000465 |

&#42; p99/p99.9 KLD are per-token percentiles over ~145K values (127 chunks × 1151 tokens per chunk). At 2304-token chunks, within-chunk KLD correlation is high: the effective tail sample for p99.9 is ~1-3 chunks, for p99 ~10-15 chunks. For ranking quants with a large gap (30%+), these metrics are reliable; for close quants (<10% in mean KLD), p99 and p99.9 carry high noise. Mean KLD is the primary metric; p95 is secondary.

At matched sizes, the allocator quant has lower mean KLD in every case.

The gap is largest at small sizes (e.g. 390 MiB: 0.119 vs 0.378, −69%) and

narrowest at the largest size (754 MiB: 0.0023 vs 0.0027, −15%). At 511 MiB

vs unsloth IQ4_NL: mean KLD 0.0267 vs 0.0453 (−41%), max KLD 1.92 vs 2.80

(−32%).

Allocator 434 MiB (mean KLD 0.105) is worse than allocator 409 MiB (0.090),

even though the allocator's proxy score was better at 434 MiB (total proxy

loss 814 vs 1151 — the assignment was optimal for the proxy). The difference

is in the token embedding tensor: at 409 MiB the allocator dropped it to

IQ4_XS, freeing budget for more important tensors; at 434 MiB it kept it in

Q5_K, spending extra budget where it mattered less. The same pattern appears

in the unsloth lineup: unsloth UD_Q2_K_XL (433.52 MiB, KLD 0.326) is worse than

the smaller unsloth UD_IQ3_XXS (409.23 MiB, KLD 0.241) — Q2_K is a known weak

type, and unsloth also made a similar upgrade mistake further up the line

(UD_Q6_K_XL → UD_Q5_K_XL on embeddings at a larger size step). Both dips are real

(verifiable by GGUF paths in evaluation logs), not swapped data. They show

that the proxy correlates with KLD but is not exact — the allocator's proxy

is very cheap compared to real KLD runs, but it is an approximation, not a

perfect substitute.

Results: wikitext-2 test (all models)

Same comparison on an unrelated corpus — all 16 models evaluated:

| Model | Size (MiB) | Mean PPL | Mean KLD | p95 KLD | p99 KLD&#42; | p99.9 KLD&#42; | Max KLD | RMS Δp | Same top p | Top flip weight |

|---|---|---|---|---|---|---|---|---|---|---|

| BF16 baseline | 1485.50 | 15.508 | — | — | — | — | — | — | — | — |

| allocator 390 | 390.00 | 16.484 | 0.1317 | 0.3638 | 0.6880 | 1.7232 | 8.9320 | 8.640 | 81.366 | 0.020899 |

| unsloth UD_IQ2_M | 390.37 | 21.811 | 0.3751 | 1.0215 | 1.9303 | 4.3372 | 15.2365 | 15.452 | 69.864 | 0.057970 |

| allocator 409 | 409.00 | 15.976 | 0.0955 | 0.2621 | 0.4978 | 1.2375 | 10.1608 | 7.449 | 83.795 | 0.015462 |

| unsloth UD_IQ3_XXS | 409.23 | 19.021 | 0.2349 | 0.6752 | 1.3078 | 3.2066 | 10.0378 | 11.987 | 76.417 | 0.035573 |

| allocator 434 | 434.00 | 16.607 | 0.1094 | 0.3123 | 0.5990 | 1.5675 | 9.1137 | 7.884 | 83.556 | 0.017057 |

| unsloth UD_Q2_K_XL | 433.52 | 20.656 | 0.3073 | 0.8828 | 1.6389 | 3.8417 | 14.4516 | 13.264 | 73.604 | 0.045124 |

| allocator 485 | 484.99 | 16.135 | 0.0414 | 0.1164 | 0.2225 | 0.5705 | 4.3733 | 4.836 | 89.341 | 0.006592 |

| unsloth UD_Q3_K_XL | 484.54 | 16.698 | 0.0860 | 0.2471 | 0.4670 | 1.1557 | 6.5099 | 7.053 | 84.754 | 0.013869 |

| allocator 511 | 510.99 | 15.906 | 0.0289 | 0.0806 | 0.1580 | 0.4358 | 2.9061 | 4.096 | 90.897 | 0.004670 |

| unsloth IQ4_NL | 510.59 | 16.186 | 0.0469 | 0.1325 | 0.2603 | 0.7065 | 4.0409 | 5.153 | 88.837 | 0.007454 |

| allocator 546 | 545.99 | 15.816 | 0.0266 | 0.0754 | 0.1493 | 0.4091 | 3.8512 | 3.873 | 91.478 | 0.004207 |

| unsloth UD_Q4_K_XL | 546.44 | 15.610 | 0.0301 | 0.0854 | 0.1581 | 0.3696 | 2.2790 | 3.993 | 90.831 | 0.004693 |

| allocator 586 | 585.99 | 15.806 | 0.0121 | 0.0339 | 0.0653 | 0.1846 | 2.1765 | 2.655 | 93.896 | 0.001941 |

| unsloth UD_Q5_K_XL | 585.75 | 15.722 | 0.0125 | 0.0340 | 0.0632 | 0.1729 | 1.7454 | 2.618 | 93.738 | 0.001965 |

| allocator 754 | 753.97 | 15.600 | 0.0026 | 0.0070 | 0.0132 | 0.0369 | 0.3462 | 1.249 | 96.695 | 0.000399 |

| unsloth UD_Q6_K_XL | 754.21 | 15.622 | 0.0028 | 0.0075 | 0.0137 | 0.0336 | 0.2571 | 1.278 | 96.739 | 0.000394 |

&#42; Same caveat as for kilo-small: ~127 chunks, ~145K per-token KLD values, effective tail sample constrained by within-chunk correlation. Mean KLD is the primary metric.

Relative ranking preserved: allocator quants have lower mean KLD at matched

sizes on an unrelated corpus. Absolute KLD values are higher (wikitext-2 is a

harder domain for this model), but the allocator vs unsloth gap remains

consistent. At 511 MiB vs unsloth IQ4_NL: mean KLD 0.0289 vs 0.0469 (−38%),

max KLD 2.91 vs 4.04 (−28%).

Notes

• The allocator is a prototype, tested here on a single small model. Results

on larger models may differ.

• Evaluation uses a single calibration corpus hold-out and one standard corpus;

results reflect these specific distributions.

• Perplexity values are per-chunk means with 2304-token chunks; they are

internally consistent but not directly comparable to PPL reported with

different chunk sizes.

• Both the allocator and unsloth quants preserve MTP/NextN tensors; quality

differences reflect type assignment strategy, not MTP handling.

Acknowledgments

• Alibaba and Qwen for the best open models that can run on consumer hardware
• Unsloth for quantized models used in comparisons and for setting a high quality bar
• michaelw9999 for ideas, inspiration, and advanced-gguf-quantizer