---

license: apache-2.0

base_model: empero-ai/Qwen3.8-9B-Distill

tags:

• gguf
• qwen3.5
• llama.cpp
• speculative-decoding
• mtp

---

Dirk-Qwen3.8-9B-GGUF

Dirk is a lean chat build of the Qwen3.8-9B model. The weights are unchanged. Only the chat template is different.

This project follows the style of peculiar-ragdoll/Dirk-Qwen3.8-27B-GGUF. It is not an official release by peculiar-ragdoll or by Qwen.

What this build changes

The stock template lets the model think for a long time. This build changes the default reasoning effort to low. The model gives short answers out of the box. You can raise the effort for one request at a time.

The base model is a reasoning distill. At the default medium effort, it wrote more than 1200 tokens of thinking for a simple question. At low effort, it wrote 69 tokens in total. The answer quality stayed the same.

The template is Sharp v22.4.1 from peculiar-ragdoll/Qwen-Sharp-Chat-Templates. This build makes one change to it. The default reasoning effort is low. The upstream default is medium.

Files

| file | size | notes |

| --- | --- | --- |

| Dirk-Qwen3.8-9B-Q4_K_M.gguf | 5.78 GB | Smallest. |

| Dirk-Qwen3.8-9B-Q5_K_M.gguf | 6.64 GB | Good balance. |

| Dirk-Qwen3.8-9B-Q6_K.gguf | 7.56 GB | Best quality for a 12 GB GPU. |

| Dirk-Qwen3.8-9B-Q8_0.gguf | 9.79 GB | Highest precision. Needs 16 GB or a GPU pool. |

| Dirk-Qwen3.8-9B-BF16.gguf | 18.41 GB | Full precision. Use it to make other quants. |

| mtp-Qwen3.8-9B-Distill-head-Q8_0.gguf | 2.43 GB | MTP head. Required for speculative decoding. |

All GGUF files keep the MTP tensors. MTP means multi-token prediction. MTP predicts several tokens at once.

The file sizes exclude the 2.43 GB head file. On a 12 GB GPU, use Q6_K or smaller with the head. Q8_0 does not fit a 12 GB GPU together with the head.

llama.cpp ignores the MTP tensors inside the main file. You must load the head file as the draft model. Without the head file, the model runs without speculative decoding.

Run it

You need llama.cpp with Vulkan support or ROCm support.

1. Download a main file and the head file.
2. Start the server.

llama-server \
  -m Dirk-Qwen3.8-9B-Q6_K.gguf \
  -md mtp-Qwen3.8-9B-Distill-head-Q8_0.gguf \
  --spec-type draft-mtp \
  -ngl 99

You can change the thinking effort for one request. Send chat_template_kwargs with the effort value.

{"chat_template_kwargs": {"reasoning_effort": "high"}}

The effort levels are low, medium, high, and xhigh. To turn thinking off, send {"chat_template_kwargs": {"enable_thinking": false}}.

llama.cpp removes the top-level reasoning_effort field. Put the effort in chat_template_kwargs.

Measured results

We measured these results on one machine. The GPU is an AMD Radeon RX 7700 XT with 12 GB. The build is llama.cpp b10705 with Vulkan. The model is the Q6_K file with the head file.

| measure | value |

| --- | --- |

| Draft acceptance | 0.55 to 0.78 |

| Eval speed | 67 to 95 tokens per second |

| Default answer, simple question | 74 completion tokens |

| No-thinking answer, simple question | 53 completion tokens |

Draft acceptance is the share of draft tokens that the model accepts.

Reproduce this build

The folder scripts contains the build tools.

• build_dirk_9b.sh runs the whole build.
• gguf_meta_swap.py changes a string value in GGUF metadata.
• chat_template_oneline_dirk9b_low.txt is the modified template.
• chat_template_sharp_v22.4.1_pristine.jinja is the upstream template. It is unmodified.

You need llama.cpp and Python 3.

./scripts/build_dirk_9b.sh model-bf16.gguf Q4_K_M Q5_K_M Q6_K Q8_0

The script needs a BF16 GGUF file of the base model. The script writes the quants into the current folder.

Model details

• Base model: empero-ai/Qwen3.8-9B-Distill
• Architecture: qwen35
• Context length: 262,144 tokens natively
• Type: text model
• License: Apache-2.0

The base model is a distillation of the Qwen3.8 teacher into the Qwen3.5-9B architecture.

Provenance

• The weights come from Empero. They are Apache-2.0.
• The template comes from peculiar-ragdoll. This build changes the default effort to low.
• No training was done. No weights were changed.