---

license: apache-2.0

language:

• en

pipeline_tag: text-generation

library_name: gguf

tags:

• instruct
• chat
• tool-calling
• qwen3
• gguf

base_model: VertexResearch/Vertex-0.6-100M-8192-Instruct-v2

---

Vertex 0.6 100M — 8192-ctx Instruct v2 (GGUF)

GGUF conversion of VertexResearch/Vertex-0.6-100M-8192-Instruct-v2

for use with llama.cpp.

Files

| File | Type |

|---|---|

| Vertex-0.6-100M-8192-Instruct-v2-BF16.gguf | Full precision |

| Vertex-0.6-100M-8192-Instruct-v2-F16.gguf | Full precision |

| Vertex-0.6-100M-8192-Instruct-v2-Q8_0.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q5_K_M.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q5_K_S.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q4_K_M.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q4_K_S.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q4_1.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q4_0.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q3_K_M.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q3_K_S.gguf | Quantized |

| Vertex-0.6-100M-8192-Instruct-v2-Q2_K.gguf | Quantized |

BF16/F16 are the recommended full-precision files (deliverable quality); the

rest are size-reduced quants for constrained deployment — this model is tiny

(97M params), so quantization mainly saves disk, not meaningfully more speed.

Model details

| | |

|---|---|

| Parameters | 96.75M (tied embeddings) |

| Architecture | Qwen3-based transformer |

| Context length | 8192 (RoPE theta 1M) |

| Chat format | ChatML (<\|im_start\|> / <\|im_end\|>) |

| Tool calling | <tool_call> JSON blocks, system-prompt function definitions |

| EOS | </s> (2) and <\|im_end\|> (6) |

Usage

llama-server -m Vertex-0.6-100M-8192-Instruct-v2-F16.gguf --host 0.0.0.0 --port 8080

or raw completion smoke test:

llama-completion -m Vertex-0.6-100M-8192-Instruct-v2-F16.gguf -p "The sky is" -n 16 --temp 0

Limitations

These models are not the most coherent yet and need more tuning: expect rambling,

repetition, and inconsistent answers, especially over longer generations.

97M parameters: multi-turn chat is much improved and tool-call syntax works, but

factual accuracy is low, reasoning is shallow, and it makes arithmetic errors.

Not for production use. Knowledge cutoff \~April 2024.

Tool calling works mechanically (correct <tool_call> format, stops cleanly)

with a single available function, but multi-step tool use is unreliable: with

more than one function available it can pick the wrong tool, and it can

hallucinate details when summarizing a tool's response rather than reporting it

accurately. Don't trust it in an unsupervised agent loop.