---

language:

- en

license: apache-2.0

base_model:

- unsloth/Qwen2.5-1.5B-Instruct

datasets:

- r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation

tags:

- qwen2.5

- qwen

- tool-calling

- function-calling

- agents

- lora

- sft

- gguf

library_name: gguf

---

Qwen2.5-1.5B Tools GGUF

The Qwen2.5-1.5B tool-calling model in GGUF format, fine-tuned

for tool calling and agent-style interactions.

Base model

This model was fine-tuned from:

unsloth/Qwen2.5-1.5B-Instruct

GGUF files

The model is provided in GGUF format at the following precisions:

| Precision | File |

|---|---|

| BF16 | Qwen2.5-1.5B-tools-BF16.gguf (original precision) |

| Q4_K_M | Qwen2.5-1.5B-tools-Q4_K_M.gguf |

| Q5_K_M | Qwen2.5-1.5B-tools-Q5_K_M.gguf |

| Q8_0 | Qwen2.5-1.5B-tools-Q8_0.gguf |

Training

Training was performed using NVIDIA NeMo AutoModel with LoRA/PEFT.

LoRA configuration

• LoRA rank (dim): 32
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target modules: *.proj

Training configuration

• Sequence length: 4096
• Micro batch size: 2
• Gradient accumulation: 32
• Effective batch size: 64
• Steps: 501
• Learning rate: 1e-5
• Weight decay: 0.01
• Optimizer: AdamW
• Loss: masked cross entropy
• Reasoning content masking: disabled

Dataset

The model was trained on the sft_tools portion of:

r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation

The dataset contains tool/agent interaction examples.

Intended use

This model is intended for:

• tool calling
• function calling
• agent workflows
• structured tool interactions
• local AI assistants
• lightweight local inference

It is not intended to be a general replacement for larger Qwen models.

GGUF versions

The model is available in GGUF format at:

• BF16
• Q8_0
• Q5_K_M
• Q4_K_M

Jetson Nano

Q4_K_M is the recommended starting point for low-memory systems such as the Jetson Nano.

For higher-memory systems, use Q5_K_M or Q8_0 where practical.

Usage with llama.cpp

Example:

llama-cli \
  -hf iromu/Qwen2.5-1.5B-tools-GGUF:Q4_K_M \
  --n-gpu-layers 99 \
  --ctx-size 1024 \
  -p "You are a home assistant. Convert user requests into JSON tool calls. User: turn on kitchen lights"

llama-server \
  -hf iromu/Qwen2.5-1.5B-tools-GGUF:Q4_K_M \
  --jinja \
  --n-gpu-layers 99 \
  --ctx-size 4096 \
  --host 0.0.0.0 \
  --port 8080

<!-- VALIDATION:BEGIN (auto-generated, do not edit) -->

Validation matrix

Tool-calling validation on the sft_tools validation split (greedy decoding, 384 max new tokens). Throughput is single-stream greedy decode, not serving throughput.

Pretrained base (unsloth/Qwen2.5-1.5B-Instruct): 8.8% exact-args match (24/274).

Fine-tuned (BF16): 0.7% exact-args match (2/274) (-8pp vs base).

• GGUF-BF16: 2/274 (0.7%) exact, 88.6 tok/s — 100% of BF16.
• GGUF-Q4_K_M: 3/274 (1.1%) exact, 142.9 tok/s — 150% of BF16.
• GGUF-Q5_K_M: 6/274 (2.2%) exact, 131.4 tok/s — 300% of BF16.
• GGUF-Q8_0: 2/274 (0.7%) exact, 115.3 tok/s — 100% of BF16.

| Model | Quant | n | Tool call emitted | Names match | Exact args match | Δ exact vs BASE | tok/s |

|---|---|---|---|---|---|---|---|

| Qwen2.5-1.5B-tools | BASE (unsloth/Qwen2.5-1.5B-Instruct) | 274 | 272/274 (99.3%) | 59/274 (21.5%) | 24/274 (8.8%) | — | 40.6 |

| Qwen2.5-1.5B-tools | BF16 | 274 | 258/274 (94.2%) | 216/274 (78.8%) | 2/274 (0.7%) | -8pp | 38.0 |

| Qwen2.5-1.5B-tools | GGUF-BF16 | 274 | 272/274 (99.3%) | 167/274 (60.9%) | 2/274 (0.7%) | -8pp | 88.6 |

| Qwen2.5-1.5B-tools | GGUF-Q4_K_M | 274 | 248/274 (90.5%) | 49/274 (17.9%) | 3/274 (1.1%) | -7.7pp | 142.9 |

| Qwen2.5-1.5B-tools | GGUF-Q5_K_M | 274 | 260/274 (94.9%) | 79/274 (28.8%) | 6/274 (2.2%) | -6.6pp | 131.4 |

| Qwen2.5-1.5B-tools | GGUF-Q8_0 | 274 | 271/274 (98.9%) | 187/274 (68.2%) | 2/274 (0.7%) | -8pp | 115.3 |

<!-- VALIDATION:END -->