---

license: apache-2.0

base_model: ai9stars/G9v3-39A5B

pipeline_tag: text-generation

tags:

• g9v3
• gguf
• llama-cpp
• cpu
• moe
• imatrix
• tool-calling
• long-context

language:

• en
• zh

---

G9v3-39A5B IQ4_XS GGUF — 16-core CPU tier

An imatrix IQ4_XS cut of ai9stars/G9v3-39A5B

(39B MoE, ~5.4B active params/token, 131K context, think/no-think modes, tool calling)

sized for the first CPU tier that can hold the big model: 16 cores / ≥32 GB RAM.

The GGUF is imatrix-quantized by linuxid10t/G9v3-39A5B-GGUF;

this repo adds the serving template and measured performance numbers.

⚠️ Read "llama.cpp requirements" below before downloading — stock llama.cpp cannot

load this architecture; you need the g9v3 fork.

What's in this repo

| File | Size | What it is |

|---|---|---|

| G9v3-39A5B-IQ4_XS.gguf | 19.4 GB | the quantized model (4.28 BPW, imatrix) |

| g9v3_chat_template_low.jinja | 12 KB | chat template — required for tool calling (see below) |

llama.cpp requirements (important)

Upstream llama.cpp has no g9v3 architecture support. Serving this GGUF requires:

1. The fork: linuxid10t/llama.cpp, branch

feature/g9v3-support

1. A required local patch: in src/llama-context.cpp, graph_max_nodes() must include

LLM_ARCH_G9V3 in the MoE branch (n_tokens * 40 budget) — otherwise the server

aborts at startup with GGML_ASSERT(obj_new) failed at any context size.

The fork build also serves plain llama-arch GGUFs (e.g. our

3B cuts) — one binary works for all.

Measured performance (8 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz with llama.cpp llama-server:

• Decode: ~13 tok/s, prefill: ~43 tok/s (a 16-core box projects ~15–20 tok/s)
• RSS: ~19.9 GiB — validated at 8192 context; a 32 GB box can go higher

(KV cache is 38 KiB/token on this model: 32K ≈ 1.2 GiB/seq, 131K ≈ 4.75 GiB/seq)

• Sanity: France question → reasoning separated, answer "The capital of France is Paris."

With only ~5.4B active params/token, a 39B-class MoE runs on plain CPU — slower than

the 3B dense cuts but far more

capable at tool-calling and multi-step work.

Run

# get the files
huggingface-cli download Itopoly/G9v3-39A5B-IQ4_XS-GGUF G9v3-39A5B-IQ4_XS.gguf --local-dir .
huggingface-cli download Itopoly/G9v3-39A5B-IQ4_XS-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve with the g9v3 fork build (see requirements above)
llama-server -m G9v3-39A5B-IQ4_XS.gguf --alias g9v3-39a5b-iq4_xs \
  -t 8 -tb 8 -c 8192 -fa off --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja

8 threads is what we validated; on a 16-core box try -t 12..16. OpenAI-compatible on

/v1/chat/completions; reasoning arrives in message.reasoning, the answer in content.

Why the chat template matters

Don't serve without g9v3_chat_template_low.jinja: the stock template drops tool-result

messages and the model re-calls the same tool forever (an observed loop bug). The bundled

template normalizes legacy tool history so multi-turn tool use works.

Notes

• Think mode: temperature=1.0, top_p=0.95; no-think mode: temperature=0.7, top_p=0.95.
• -fa off recommended — flash-attention is slower on CPU.
• No idle eviction: model and prompt cache stay resident; repeat requests on the same

prefix are near-instant.

• The MoE family ladder: 3B (this repo's 3B siblings), 39A5B (this repo). Middle rungs

(20A5B / 10A5B expert-pruned cuts) are planned — see our W4A16 vLLM repo for the GPU tier:

Itopoly/G9v3-39A5B-W4A16.

Credits

• Original model: ai9stars/G9v3-39A5B (Apache-2.0)
• GGUF quantization (imatrix): linuxid10t/G9v3-39A5B-GGUF
• CPU-tier validation, template, and packaging: Itopoly

(GPU serving: our W4A16 quant)