---

pipeline_tag: text-generation

base_model:

• ai9stars/G9v3-3B

license: apache-2.0

license_name: apache-license-2.0

library_name: llama-cpp

tags:

• g9v3
• llama
• text-generation
• gguf
• quantized

languages:

• en
• zh

---

G9v3-3B GGUF

GGUF quantized versions of ai9stars/G9v3-3B, a compact 3B-parameter Llama-architecture language model supporting English and Chinese, with a 128K context window, tool-calling, and a built-in thinking/reasoning mode.

Model Overview

G9v3-3B is a lightweight text-generation model built on the Llama architecture. Despite its small size (~3B parameters), it supports a 131,072-token context window (128K) thanks to a high RoPE θ of 5,000,000. The model uses Grouped-Query Attention (GQA) with 2 key-value heads across 16 attention heads for efficient inference, and includes special tokens for thinking/reasoning (<|thought_begin|> / <|thought_end|>, /think, /no_think) and tool calling (<function>, <|tool_call|>, etc.).

The model uses a ChatML-style conversation format with <|im_start|> / <|im_end|> delimiters.

Model Architecture

| Property | Value |

| :----------------------- | :--------- |

| Architecture | Llama |

| Parameters | ~3B |

| Hidden Size | 2048 |

| Intermediate Size | 6144 |

| Layers | 52 |

| Attention Heads | 16 |

| KV Heads | 2 (GQA) |

| Head Dimension | 128 |

| Context Length | 131,072 |

| Vocabulary Size | 130,560 |

| RoPE Theta | 5,000,000 |

| Original Precision | bfloat16 |

| Supported Languages | en, zh |

Available GGUF Files

| File | Quantization | Size | Use Case |

| :------------------------- | :----------- | :---- | :----------------------------------------- |

| g9v3-3b-bf16.gguf | BF16 | 5.6 GB | Max precision, reference model |

| g9v3-3b-Q8_0.gguf | Q8_0 | 3.0 GB | Near-lossless, good speed/quality balance |

| g9v3-3b-Q6_K.gguf | Q6_K | 2.3 GB | Very high quality, recommended for quality |

| g9v3-3b-Q5_K_M.gguf | Q5_K_M | 2.0 GB | High quality, balanced |

| g9v3-3b-Q5_K_S.gguf | Q5_K_S | 2.0 GB | High quality, slightly smaller than M |

| g9v3-3b-Q4_K_M.gguf | Q4_K_M | 1.8 GB | Good quality, recommended default |

| g9v3-3b-Q4_K_S.gguf | Q4_K_S | 1.7 GB | Smaller, acceptable quality |

| g9v3-3b-Q4_0.gguf | Q4_0 | 1.7 GB | Legacy quant, fastest inference |

> Recommended: Q4_K_M or Q5_K_M offer the best quality-to-size trade-off for most use cases.

Usage

llama.cpp CLI

./llama-cli \
  -m g9v3-3b-Q4_K_M.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 0.9 --top-p 0.95

llama-server (OpenAI-compatible API)

./llama-server \
  -m g9v3-3b-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080

Thinking Mode

The model supports a thinking/reasoning mode controlled via special tokens. Use /think to enable extended reasoning or /no_think to disable it. When thinking is enabled, the model outputs its reasoning between <|thought_begin|> and <|thought_end|> tokens before providing the final answer.

Tool Calling

The model supports function/tool calling via XML-style <function> tags. Tool definitions are injected into the system prompt, and the model responds with <function name="..."> blocks containing <param> elements.

Generation Parameters

Recommended parameters from the original model's generation_config.json:

| Parameter | Value |

| :------------ | :---- |

| Temperature | 0.9 |

| Top-P | 0.95 |

| BOS Token ID | 0 |

| EOS Token IDs | 1, 130073 |

| Pad Token ID | 1 |

Quantization

These GGUF files were created from the BF16 source model using llama-quantize from the llama.cpp project.

Acknowledgements

• Original model: ai9stars/G9v3-3B
• Quantization tool: llama.cpp

License

Apache License 2.0