---

language:

- en

license: apache-2.0

base_model:

- iromu/Qwen3-0.6B-tools

datasets:

- r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation

tags:

- qwen3

- qwen

- tool-calling

- function-calling

- agents

- sft

- lora

- gguf

library_name: gguf

---

Qwen3-0.6B-tools GGUF

The Qwen3-0.6B tool-calling fine-tune in GGUF format, for llama.cpp-style

runtimes. The model was supervised-fine-tuned with LoRA so it reliably emits

OpenAI-style tool calls for agentic workflows. The full BF16 model and its

validation table live at iromu/Qwen3-0.6B-tools.

Base model

This model is the GGUF conversion of:

iromu/Qwen3-0.6B-tools

— itself a LoRA fine-tune of Qwen/Qwen3-0.6B.

GGUF files

The model is provided in GGUF format at the following precisions:

| Precision | File | Notes |

|---|---|---|

| BF16 | Qwen3-0.6B-tools-BF16.gguf | original precision |

| Q4_K_M | Qwen3-0.6B-tools-Q4_K_M.gguf | lowest VRAM / fastest |

| Q5_K_M | Qwen3-0.6B-tools-Q5_K_M.gguf | compact |

| Q6_K | Qwen3-0.6B-tools-Q6_K.gguf | between Q5_K_M and Q8_0 in size |

| Q8_0 | Qwen3-0.6B-tools-Q8_0.gguf | fidelity pick — at parity with the BF16 model |

The uniform quants above are joined by an imatrix-guided ("Dynamic") family,

UD-Q2_K_XL through UD-Q8_K_XL. These allocate bits per tensor by activation

importance rather than uniformly, so a file lands near the size of its nominal tier

with better fidelity:

| Precision | File | Notes |

|---|---|---|

| UD-Q2_K_XL | Qwen3-0.6B-tools-UD-Q2_K_XL.gguf | imatrix-guided; below the uniform Q4 floor |

| UD-Q3_K_XL | Qwen3-0.6B-tools-UD-Q3_K_XL.gguf | imatrix-guided |

| UD-Q4_K_XL | Qwen3-0.6B-tools-UD-Q4_K_XL.gguf | imatrix-guided; same nominal tier as Q4_K_M |

| UD-Q5_K_XL | Qwen3-0.6B-tools-UD-Q5_K_XL.gguf | imatrix-guided |

| UD-Q6_K_XL | Qwen3-0.6B-tools-UD-Q6_K_XL.gguf | imatrix-guided |

| UD-Q8_K_XL | Qwen3-0.6B-tools-UD-Q8_K_XL.gguf | imatrix-guided; highest precision |

The importance matrix behind the UD-* family was calibrated on the SFT train

split under this model's chat template — never on the validation split the

validation table below scores on.

On this model the exact-args fidelity (measured on the sft_tools validation

split; full numbers in the validation table below) is not a smooth ramp in quant

size. Q4_K_M drops to about a third of the BF16 rate; Q5_K_M and Q6_K sit in

the same ~54% band, so Q6_K costs more bytes for no fidelity gain over

Q5_K_M; and Q8_0 is the fidelity pick — it reaches parity with the BF16

conversion (~66% exact args) and is the quant to use when the exact tool

arguments matter.

Training

Supervised fine-tuning with LoRA, run with NeMo-RL, assistant-turn-masked

cross-entropy over the tool-calling logs.

• LoRA: dim 32, alpha 32, dropout 0.05, target modules *.proj

(q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)

• Max sequence length: 4096
• Precision: bfloat16
• Optimizer: AdamW, lr 5e-5, weight decay 0.01
• Schedule: 15-step linear warmup, then cosine decay to 1e-6
• Global batch size: 64 (micro batch 2)
• Training steps: 336 (validation every 84)
• Loss masked to assistant turns via the chat template's generation keyword

Dataset

Training used the sft_tools split of the

r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation dataset (OpenAI messages

format; assistant turns carry tool_calls).

Intended use

• Structured tool / function calling in llama.cpp-style runtimes
• Agent-style multi-step interactions
• Small-footprint on-device or edge deployment

It is not intended to be a general replacement for larger Qwen models.

Sibling artifacts

• BF16 model + validation table: iromu/Qwen3-0.6B-tools
• NVFP4 checkpoint: iromu/Qwen3-0.6B-tools-NVFP4

Usage

Load a GGUF quant with llama.cpp:

llama-cli -hf iromu/Qwen3-0.6B-tools-GGUF:Q4_K_M

Or with the llama.cpp Python bindings:

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_path="iromu/Qwen3-0.6B-tools-GGUF",
    filename="Qwen3-0.6B-tools-Q5_K_M.gguf",
    n_gpu_layers=99,
)

<!-- VALIDATION:BEGIN (auto-generated, do not edit) -->

Validation

| Model | Quant | Cap | n | Tool call emitted | Names match | Exact args match | Δ exact vs BASE (gated) |

|---|---|---|---|---|---|---|---|

| Qwen/Qwen3-0.6B | BF16 | 384 tok | 274 | 48/274 (17.5%) | 9/274 (3.3%) | 2/274 (0.7%) | (no predecessor) |

| Qwen3-0.6B-tools | GGUF-BF16 | 384 tok | 274 | 272/274 (99.3%) | 228/274 (83.2%) | 175/274 (63.9%) | +63.1pp |

| Qwen3-0.6B-tools | GGUF-Q4_K_M | 384 tok | 274 | 272/274 (99.3%) | 150/274 (54.7%) | 92/274 (33.6%) | +32.8pp |

| Qwen3-0.6B-tools | GGUF-Q5_K_M | 384 tok | 274 | 272/274 (99.3%) | 190/274 (69.3%) | 150/274 (54.7%) | +54pp |

| Qwen3-0.6B-tools | GGUF-Q6_K | 384 tok | 274 | 272/274 (99.3%) | 196/274 (71.5%) | 148/274 (54.0%) | +53.3pp |

| Qwen3-0.6B-tools | GGUF-Q8_0 | 384 tok | 274 | 272/274 (99.3%) | 230/274 (83.9%) | 181/274 (66.1%) | +65.3pp |

| Qwen3-0.6B-tools | GGUF-UD-Q2_K_XL | 384 tok | 274 | 24/274 (8.8%) | 2/274 (0.7%) | 2/274 (0.7%) | +0pp |

| Qwen3-0.6B-tools | GGUF-UD-Q3_K_XL | 384 tok | 274 | 259/274 (94.5%) | 81/274 (29.6%) | 67/274 (24.5%) | +23.7pp |

| Qwen3-0.6B-tools | GGUF-UD-Q4_K_XL | 384 tok | 274 | 272/274 (99.3%) | 189/274 (69.0%) | 136/274 (49.6%) | +48.9pp |

| Qwen3-0.6B-tools | GGUF-UD-Q5_K_XL | 384 tok | 274 | 271/274 (98.9%) | 191/274 (69.7%) | 145/274 (52.9%) | +52.2pp |

| Qwen3-0.6B-tools | GGUF-UD-Q6_K_XL | 384 tok | 274 | 272/274 (99.3%) | 227/274 (82.8%) | 181/274 (66.1%) | +65.3pp |

| Qwen3-0.6B-tools | GGUF-UD-Q8_K_XL | 384 tok | 274 | 272/274 (99.3%) | 224/274 (81.8%) | 172/274 (62.8%) | +62pp |

Measured on the sft_tools held-out validation split, greedy decoding. Throughput is single-stream greedy decode, not serving throughput.

suppressed in the Δ column means no comparison was admissible, which is a statement about the provenance of the two rows and not a measurement of no change. The reason is printed in the cell.

The *UD- rows are imatrix-guided (Dynamic) quants, distinct from the uniform quants: their per-tensor layout allocates bits by activation importance, and the importance matrix was calibrated on the SFT train** split under the tag's pinned chat template — never on the validation split this table scores on.

<!-- VALIDATION:END -->