---

license: apache-2.0

base_model: ai9stars/G9v3-3B

pipeline_tag: text-generation

tags:

• g9v3
• gguf
• llama-cpp
• cpu
• tool-calling
• long-context

language:

• en
• zh

---

G9v3-3B Q3_K_M GGUF — 4-core CPU tier

A CPU-tier-validated Q3_K_M cut of ai9stars/G9v3-3B

(dense ~3B, LlamaForCausalLM, 131K context, think/no-think modes, XML tool calling),

packaged for llama.cpp on low-core machines. The GGUF itself is quantized by

mradermacher/G9v3-3B-GGUF;

this repo adds the serving template and measured performance numbers for the

4-core tier.

What's in this repo

| File | Size | What it is |

|---|---|---|

| g9v3-3B.Q3_K_M.gguf | 1.5 GB | the quantized model |

| g9v3_chat_template_low.jinja | 12 KB | chat template — required for tool calling (see below) |

Measured performance (4 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz run at 4 threads to simulate a 4-core box,

with llama.cpp llama-server:

• Decode: ~25 tok/s (think-mode ~14 tok/s) — comfortably above the ~10 tok/s

interactive floor for a reasoning model

• Prompt processing: ~55 tok/s — a 5.5K-token prompt costs ~100 s once; the

prefix cache makes repeat requests ~0.6 s

• Sanity check: reasoning is separated from the answer, tool calls emit structured

tool_calls, and legacy OpenAI function_call history is normalized by the template

RAM need: ≥6 GB. Context memory: KV cache is 52 KiB/token on this model — 8192 ctx

fits an 8 GB box, 32768 ctx fits 16 GB.

Run with llama.cpp

# get the files
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3-3B.Q3_K_M.gguf --local-dir .
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve (OpenAI-compatible on /v1/chat/completions, model name: g9v3-3b)
llama-server -m g9v3-3B.Q3_K_M.gguf --alias g9v3-3b \
  -t 4 -tb 4 -c 8192 --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja

Reasoning text arrives in message.reasoning, the answer in content.

Why the chat template matters

Don't serve this GGUF without g9v3_chat_template_low.jinja: the stock template drops

tool-result messages, and the model re-calls the same tool forever (an observed loop

bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

Notes

• No-think mode: temperature=0.7, top_p=0.95; think mode: temperature=1.0, top_p=0.95.
• Prefill is the bottleneck on CPU — keep prompts ≤ 1–2K tokens, or accept a one-time

~2 min first hit per distinct large prefix (cached afterwards).

• Tool calling works (verified single-turn + both tool-history formats), but tool-call

quality on a 3B is tier-limited; the 39B family models are the tool-heavy choice.

Credits

• Original model: ai9stars/G9v3-3B (Apache-2.0)
• GGUF quantization: mradermacher/G9v3-3B-GGUF
• CPU-tier validation, template, and packaging: itopoly