---

license: apache-2.0

library_name: llama.cpp

pipeline_tag: text-generation

base_model:

• Kwaipilot/KAT-Coder-V2.5-Dev

base_model_relation: quantized

quantized_by: fooooooooooooooooooooooooooooooooooooooooo

tags:

• gguf
• llama.cpp
• quantized
• imatrix
• text-generation
• code
• coding
• agentic-coding
• mixture-of-experts
• moe
• qwen3.5
• qwen3.6

---

KAT-Coder-V2.5-Dev — Mixed Quantization

A mixed-quantization build of KAT-Coder-V2.5-Dev, aiming to balance model size and inference performance. The IQ4_XS quantization rounds out at about 19GB, and can comfortably fit on 22-24GB cards. On my 16GB vram 7800XT this works out to about 20+ tokens/s with 32k context. The IQ3_XXS was created to fit within a 16GB card, and I've managed to hit 46t/s.

Quantization Details

The model uses a mixture of IQ3_XXS, IQ4_XS, Q6_K and Q8_0, with selected tensors kept at higher precision to help preserve model quality. A self-calibrated imatrix was used.

|Component|IQ4_XS|IQ3_XXS |

| ------------- | ------------- |------------- |

|size| 19.5GB | 15GB|

|ffn_gate_exps, ffn_up_exps (all 40 layers) |IQ4_XS| IQ3_XXS|

|ffn_down_exps, layers 0-3 and 36-39|IQ4_XS|IQ4_XS|

|ffn_down_exps, the other 32 layers|IQ4_XS| IQ3_XXS|

|Embeddings|Q8_0|Q8_0|

|Output|Q8_0|Q8_0|

|Attention|Q6_K|Q6_K|

|ssm_out|Q6_K|Q6_K|

|Shared expert|Q6_K|Q6_K|

For more information check out the gguf dumps:

• IQ4_XS
• IQ3_XXS

Benchmarks (WIP)

• Sometimes the model will understand the problem but the code that comes out isn't the same. So I tried using a system prompt to see if it will rectify it (and it does to an extent). The system prompt used was "After writing the code, trace the first example through it and confirm the output matches."
• Methodology: Every problem in each test set are given three runs each to look for issues/variance in how the model solves them (e.g. HumanEval with 164 problems * 3 = 492)
• All models are tested at 0.7 Temp and 0.9 top-p.

Human Eval

|Model| Pass Rate | Mean Tokens | Doubt/Problem | Doubt median/p90/max | Spirals | Pass: calm / doubting | Doubt-free | Hit limit | Tok/s |

| ------------- |------------- |------------- |------------- |------------- |------------- |------------- |------------- |------------- |-------------|

|IQ4_XS (system prompt)| 96% | 489 | 0.46 | 0 / 1 / 12 | 1% | 96% / 93% | 82% | 1% | 25.9 |

|IQ4_XS (base)| 92% | 286 | 0.33 | 0 / 0 / 22 | 1% | 95% / 73% | 90% | 0% | 26.5 |

|IQ3_XXS (system prompt)| 93% | 522 | 0.59 | 0 / 1 / 32 | 1% | 96% / 82% | 82% | 1% | 37.0 |

|IQ3_XXS (base)| 92% | 348 | 0.60 | 0 / 1 / 39 | 1% | 96% / 78% | 79% | 1% | 74.0|

mbppplus

|Model| Pass Rate | Mean Tokens | Doubt/Problem | Doubt median/p90/max | Spirals | Pass: calm / doubting | Doubt-free | Hit limit | Tok/s |

| ------------- |------------- |------------- |------------- |------------- |------------- |------------- |------------- |------------- |-------------|

|IQ4_XS (system prompt)| 79% | 306 | 0.74 | 0 / 1 / 44 | 2% | 81% / 68% | 81% | 1% | 24.4 |

|IQ4_XS (base)| 78% | 201 | 0.68 | 0 / 1 / 59 | 2% | 81% / 65% | 83% | 1% | 26.4 |

|IQ3_XXS (system prompt)| 79% | 316 | 1.00 | 0 / 2 / 47 | 3% | 83% / 64% | 80% | 2% | 45.2 |

|IQ3_XXS (base)| 77% | 226 | 0.73 | 0 / 1 / 38 | 2% | 81% / 61% | 81% | 1% | 44.5 |

Rationale

After extensive testing, this configuration appeared to offer a good balance between file size and model performance. Higher precision is retained for selected components, while the majority of the model uses less bits for a more compact footprint. Logically the IQ_4XS should be better but if you have a 16GB card this can cut your token generation speed in half. The IQ3_XXS seemed to do as well as far as I can tell, but in benchmarks it failed more times due to doubt spirals.

Notes

• These are a quantized derivatives of the original model, not a new base model.
• Quantization settings reflect my own testing and trade-offs.
• The imatrix was focused mostly on webdev, python and markdown documentation.
• Testing is still underway and I'm trying to get a tangible benchmark.
• For the original model's capabilities, intended use, and licensing, refer to the original model repository.