---

tags:

• qwen
• qwen3_5_moe
• gguf
• mtp
• text-to-text
• conversational
• text-generation-inference

base_model:

• Kwaipilot/KAT-Coder-V2.5-Dev
• unsloth/Qwen3.6-35B-A3B-MTP-GGUF

notes:

• based on GGUF format quantizations by bartowski and unsloth

---

<div style="

border: 2px outset #90EE90;

background-color: #f0f0f0;

padding: 10px;

border-radius: 5px;

display: inline-block;

font-family: sans-serif;

">

2026-08-17 : added a quick [perplexity comparison][garage-ppl-tests] for recent (August 2026) Qwen3.5 35B A3B-alike models, spoiler:

* it is entirely a personal preference, but for coding-related tasks I would go for Kwaipilot/KAT-Coder-V2.5-Dev or OrionLLM/GRM-3.2-Sky.

</div>

<br>

<br>

About

What you probably want instead is one of the [APEX][apex] gguf builds by [gbuzhf][gbuzhf]:

* [gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF][gbuzhf]

However, sometimes you want to see if a more classic quantization will be more suitable for your specific task, or simply want to fit (or use) a specific vram target.

In this case, "one can't go wrong with [bartowski][bartowski] builds" (c) reddit -- except for these are sadly missing any draft models,

and a 20-30 %% gain in inference speed is not something to easily give up on.

Hence, I'll be adding here some "franken-merges" of original [bartowski builds][bartowski] with 20 MTP layers added to it from [unsloth/Qwen3.6-35B-A3B-MTP-GGUF][unsloth] model

using a [conversion script] made by [buzz][buzz].

All uploaded models were tested and add a ~20-30 % gain in inference speed compared to original ("ungrafted") models on AMD hardware.

(It is strongly recommended to use at most q8_0 quantizations for KV cache if your hardware supoorts it, especially if it has a chance to be more performant on it.)

llama.cpp invocation examples

This likely won't be too bad as a general suggestion -- but can be improved further by tailoring for a specific vram amount or device type; some examples are below:

llama-server  \
--model ${GGUF_FILENAME}  \
--flash-attn on  \
--n-gpu-layers-draft all  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

NB. For all of the examples below we assume a single user who mostly works on sequential and related tasks; your work pattern might be different, so check what works best for you.

llama.cpp invocation examples - find optimal mtp length

As for --spec-draft-n-max 4 setting above -- the exact number would obviously depend on your GPU model.

You might want to do a log scale search by trying

* spec-draft-n-max := 1,2,4,8,... (unlikely more) while the inference speed grows or stays the same,

* and then using binary search to narrow down the exact value that works best for you on your typical tasks;

* for a quick control check, your server log will show something like:

* nb: showing results for the same model running two consequitive coding tasks

[37813] 812.35.434.815 I slot launch_slot_: id  0 | task 56272 | processing task, is_child = 0
[37813] 812.37.748.230 I slot print_timing: id  0 | task 56272 | prompt eval time =     310.31 ms /   177 tokens (    1.75 ms per token,   570.40 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |        eval time =    2002.97 ms /   400 tokens (    5.01 ms per token,   199.70 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |       total time =    2313.28 ms /   577 tokens
[37813] 812.37.748.235 I slot print_timing: id  0 | task 56272 |    graphs reused =      52595
[37813] 812.37.748.238 I slot print_timing: id  0 | task 56272 | draft acceptance = 0.95482 (  317 accepted /   332 generated), mean len =  4.82
[37813] 812.37.749.091 I slot      release: id  0 | task 56272 | stop processing: n_tokens = 29120, truncated = 0

[37813] 812.38.114.222 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.988 (> 0.100 thold), f_keep = 1.000
[37813] 812.38.115.131 I slot launch_slot_: id  0 | task 56358 | processing task, is_child = 0
[37813] 812.38.767.224 I slot print_timing: id  0 | task 56358 | prompt eval time =     440.40 ms /   341 tokens (    1.29 ms per token,   774.29 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |        eval time =     211.57 ms /    20 tokens (   10.58 ms per token,    94.53 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |       total time =     651.97 ms /   361 tokens
[37813] 812.38.767.229 I slot print_timing: id  0 | task 56358 |    graphs reused =      52603
[37813] 812.38.767.232 I slot print_timing: id  0 | task 56358 | draft acceptance = 0.33333 (   12 accepted /    36 generated), mean len =  2.33

As you can see, we could have also tried --spec-draft-n-max 5 and see if that would improve the benchmarks: basically as long as "mean len" frequently shows values

which are greater than your spec-draft-n-max parameter -- you can still increment it by at least 1 and see if it improves your statistics (and waiting times).

Also, in some scenarios there could be some use for additional settings like --spec-draft-p-min 0.5.

I am no expert, but in my tests I had some luck with first determining the optimal --spec-draft-n-max value,

and then seeing if raising up the acceptance min level (and/or the "split" values such as --spec-draft-p-split) will improve things a little more.

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf

* this fits 24 GiB VRAM due to using quantized KV cache

NB: below, you would want to either _disable_ reasoning (to make response times shorter for the price of sometimes less elaborate outcome),

or _enable_ reasoning, in which case --reasoning-preserve option might speed up things for you for certain work patterns.

(And so you would normally want to omit either --chat-template-kwargs or --reasoning-preserve entries below; however, if you just leave them as they are, it would still work.)

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf  \
--no-mmproj  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--reasoning auto \
--chat-template-kwargs '{"enable_thinking":false}'  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_NL.bartowski.mtp.q8_k_xl.gguf

* this shall also fit 24 GiB VRAM by using quantized KV cache and a slightly reduced context window size ;

* for VRAM <= 16 GiB -- some further sacrifices are likely to be made, such as -cmoe, and possibly --n-gpu-layers auto

* for mini-pc devices with integrated gpu memory, you would probably have enough ram,

but you'd likely want to decrease --spec-draft-n-max -- somewhere down to 1 or 2,

depending on your hardware (your mileage may vary, see above)

(Also see the note above on --reasoning-preserve option)

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_NL.bartowski.mtp.gguf  \
--no-mmproj  \
--ctx-size 200000  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--chat-template-kwargs '{"enable_thinking":true}'  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_S.bartowski.mtp.gguf

* this shall still fit 24 GiB VRAM by using quantized KV cache and a substantially reduced context window size ;

* for mini-pc devices with integrated gpu memory, you would probably have enough ram,

but you'd likely want to decrease --spec-draft-n-max -- somewhere down to 1 or 2,

depending on your hardware (your mileage may vary, see above)

(Also see the note above on --reasoning-preserve option)

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_S.bartowski.mtp.gguf  \
--no-mmproj  \
--ctx-size 40960  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--reasoning auto \
--reasoning-preserve  \
--chat-template-kwargs '{"enable_thinking":true}'  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

[apex]: https://github.com/localai-org/apex-quant

[gbuzhf]: https://huggingface.co/gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF

[bartowski]: https://huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF

[unsloth]: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf

[buzz]: https://gist.github.com/buzz/1c439684d5e3f36492ae9f64ef7e3f67

[garage-ppl-tests]: https://huggingface.co/thread13/endless-frontier_BigBang-v1-GGUF-stub/blob/main/perplexity_comparison.md