---

license: apache-2.0

base_model: ai9stars/G9v3-3B

pipeline_tag: text-generation

tags:

• g9v3
• gguf
• llama-cpp
• cpu
• tool-calling
• long-context

language:

• en
• zh

---

G9v3-3B Q4_K_M GGUF — 6-core CPU tier

A CPU-tier-validated Q4_K_M cut of ai9stars/G9v3-3B

(dense ~3B, LlamaForCausalLM, 131K context, think/no-think modes, XML tool calling),

packaged for llama.cpp on mid-range machines. The GGUF is quantized by

mradermacher/G9v3-3B-GGUF;

this repo adds the serving template and measured performance numbers for the

6-core tier. Companion repo: Itopoly/G9v3-3B-Q3_K_M-GGUF

(the 4-core tier cut).

What's in this repo

| File | Size | What it is |

|---|---|---|

| G9v3-3B.Q4_K_M.gguf | 1.8 GB | the quantized model (4.9 BPW) |

| g9v3_chat_template_low.jinja | 12 KB | chat template — required for tool calling (see below) |

Measured performance (6 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz run at 6 threads, with llama.cpp llama-server:

• Decode: ~25 tok/s at short context (~15 tok/s at 4K ctx, ~10 tok/s at 8K ctx —

CPU decode degrades ~5 ms/token per 1K of context)

• Prefill: ~73 tok/s — a 5.5K-token prompt costs ~75–90 s once; the prefix cache

makes repeat requests near-instant

• A/B vs Q3_K_M at identical settings: Q3 ≈ 26 tok/s, Q4 ≈ 25 tok/s — on a 3B,

quant level barely moves decode; the bandwidth difference is too small. Q4 wins on

quality (visibly fewer 3-bit artifacts, coherent reasoning, correct "Paris" sanity

answer). Threads matter more than quant on a 3B.

• -fa OFF recommended — flash-attention is slower on CPU

RAM need: ≥10 GB free for 32K context (KV cache is 52 KiB/token on this model);

131K context fits a 12 GB box.

Run with llama.cpp

# get the files
huggingface-cli download Itopoly/G9v3-3B-Q4_K_M-GGUF G9v3-3B.Q4_K_M.gguf --local-dir .
huggingface-cli download Itopoly/G9v3-3B-Q4_K_M-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve (OpenAI-compatible on /v1/chat/completions, model name: g9v3-3b-q4_k_m)
llama-server -m G9v3-3B.Q4_K_M.gguf --alias g9v3-3b-q4_k_m \
  -t 6 -tb 8 -c 32768 -fa off --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja

Reasoning text arrives in message.reasoning, the answer in content.

Why the chat template matters

Don't serve this GGUF without g9v3_chat_template_low.jinja: the stock template drops

tool-result messages, and the model re-calls the same tool forever (an observed loop

bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

Notes

• No-think mode: temperature=0.7, top_p=0.95; think mode: temperature=0.9, top_p=0.95.
• Prefill is the bottleneck on CPU — keep prompts ≤ 1–2K tokens, or accept a one-time

~75–90 s first hit per distinct large prefix (cached afterwards).

• No idle eviction: the model and prompt cache stay resident, so second requests on the

same prefix are near-instant.

• Tool calling works (verified single-turn + both tool-history formats), but tool-call

quality on a 3B is tier-limited — the 39B family models are the tool-heavy choice.

Credits

• Original model: ai9stars/G9v3-3B (Apache-2.0)
• GGUF quantization: mradermacher/G9v3-3B-GGUF
• CPU-tier validation, template, and packaging: Itopoly

(see also our Q3_K_M 4-core cut)