---

language: en

license: apache-2.0

base_model: Qwen/Qwen3-0.6B

datasets:

• r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation

tags:

• qwen3
• tool-calling
• function-calling
• agents
• lora
• gguf

library_name: gguf

---

Qwen3-0.6B Tools GGUF

The Qwen3-0.6B tool-calling model in GGUF format, fine-tuned with

LoRA for tool calling and agent-style interactions.

Base model

This model was fine-tuned from:

Qwen/Qwen3-0.6B

GGUF files

The model is provided in GGUF format at the following precisions:

| Precision | File |

|---|---|

| BF16 | Qwen3-0.6B-tools-BF16.gguf (original precision) |

| Q4_K_M | Qwen3-0.6B-tools-Q4_K_M.gguf |

| Q5_K_M | Qwen3-0.6B-tools-Q5_K_M.gguf |

| Q8_0 | Qwen3-0.6B-tools-Q8_0.gguf |

Training

Training was performed using NVIDIA NeMo AutoModel with LoRA/PEFT.

LoRA configuration

• LoRA dimension: 32
• LoRA alpha: 32
• Dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training configuration

• Max sequence length: 4096
• Learning rate: 5e-5
• Weight decay: 0.01
• Global batch size: 64 (micro batch 2 x 32 accumulation)
• Training steps: 336
• Mixed precision: bf16

Dataset

Training used the sft_tools split of the

r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation dataset.

Intended use

• Structured tool/function calling
• Agent-style multi-step interactions
• Small-footprint on-device or edge deployment

It is not intended to be a general replacement for larger Qwen models.

GGUF versions

The model is available in GGUF format at:

• BF16
• Q4_K_M
• Q5_K_M
• Q8_0

Usage

Run the model with llama.cpp:

llama-cli -hf iromu/Qwen3-0.6B-tools-GGUF:Q4_K_M

The BF16 GGUF file can be quantized locally to other GGUF

precisions with llama-quantize if needed.

<!-- VALIDATION:BEGIN (auto-generated, do not edit) -->

Validation matrix

Tool-calling validation on the sft_tools validation split (greedy decoding, 384 max new tokens). Throughput is single-stream greedy decode, not serving throughput.

Pretrained base (Qwen/Qwen3-0.6B): 1.1% exact-args match (3/274).

Fine-tuned (BF16): 66.0% exact-args match (33/50) (+64.9pp vs base).

• GGUF-BF16: 18/50 (36.0%) exact, 84.4 tok/s — 55% of BF16.
• GGUF-Q4_K_M: 9/50 (18.0%) exact, 98.5 tok/s — 27% of BF16.
• GGUF-Q5_K_M: 22/50 (44.0%) exact, 99.9 tok/s — 67% of BF16.
• GGUF-Q8_0: 15/50 (30.0%) exact, 96.1 tok/s — 45% of BF16.

| Model | Quant | n | Tool call emitted | Names match | Exact args match | Δ exact vs BASE | tok/s |

|---|---|---|---|---|---|---|---|

| Qwen3-0.6B-tools | BASE (Qwen/Qwen3-0.6B) | 274 | 51/274 (18.6%) | 4/274 (1.5%) | 3/274 (1.1%) | — | 35.5 |

| Qwen3-0.6B-tools | BF16 | 50 | 49/50 (98.0%) | 43/50 (86.0%) | 33/50 (66.0%) | +64.9pp | 33.3 |

| Qwen3-0.6B-tools | GGUF-BF16 | 50 | 50/50 (100.0%) | 27/50 (54.0%) | 18/50 (36.0%) | +34.9pp | 84.4 |

| Qwen3-0.6B-tools | GGUF-Q4_K_M | 50 | 50/50 (100.0%) | 14/50 (28.0%) | 9/50 (18.0%) | +16.9pp | 98.5 |

| Qwen3-0.6B-tools | GGUF-Q5_K_M | 50 | 50/50 (100.0%) | 31/50 (62.0%) | 22/50 (44.0%) | +42.9pp | 99.9 |

| Qwen3-0.6B-tools | GGUF-Q8_0 | 50 | 50/50 (100.0%) | 24/50 (48.0%) | 15/50 (30.0%) | +28.9pp | 96.1 |

<!-- VALIDATION:END -->