---

base_model: ai9stars/G9v3-39A5B

license: apache-2.0

language:

- en

- zh

pipeline_tag: text-generation

tags:

- gguf

- llama.cpp

- quantized

- g9v3

- moe

- long-context

- tool-calling

---

G9v3-39A5B GGUF

This repository contains unofficial BF16 and Q4_K_M GGUF conversions of

ai9stars/G9v3-39A5B, a preview

Mixture-of-Experts language model from AI9Stars.

The conversion was created and tested with a llama.cpp implementation of the

G9v3 architecture available on the

feature/g9v3-support

branch. Until G9v3 support is merged upstream, standard llama.cpp builds will

not load this model.

Available files

| File | Format | Size | SHA-256 |

| --- | --- | ---: | --- |

| G9v3-39A5B-BF16.gguf | BF16 | 72.63 GiB (77,988,959,200 bytes) | 016c0c8a708fc5d4e2378df1eb9b5f089d7e91b581914e2565fb819bcebe9196 |

| G9v3-39A5B-Q4_K_M.gguf | Q4_K_M | 21.94 GiB (23,559,222,240 bytes) | 0c28e98eef634ca2304a492c7b2c92d8599189ae4409177f7edf50da5b082eb0 |

Choose the file according to your hardware and quality requirements:

| Format | Description | Recommended system RAM |

| --- | --- | ---: |

| BF16 | Highest-fidelity GGUF. Source BF16 tensors are preserved, along with required F32 tensors. | 96 GiB or more |

| Q4_K_M | Mixed K-quant that keeps selected tensors at higher precision; practical for local CPU inference. | 32 GiB or more |

The memory recommendations include headroom for the runtime, context, and

operating system. Actual requirements vary with context size and llama.cpp

settings.

Model details

| Property | Value |

| --- | --- |

| Architecture | G9v3ForCausalLM, sparse MoE |

| Parameters | Approximately 39B total / 5B active per token |

| Layers | 38 (1 dense layer followed by 37 MoE layers) |

| Hidden size | 2,048 |

| Attention | 32 query heads, 2 key/value heads, head dimension 128 |

| Experts | 320 routed experts, 32 selected per token, plus 1 shared expert |

| Expert intermediate size | 512 |

| Vocabulary | 130,560 tokens |

| Maximum context | 131,072 tokens |

| Languages | English and Chinese |

| Features | Think/No-Think modes, tool calling, long context |

The base model is a preview release. Refer to the

ai9stars/G9v3-39A5B model card

for the original description and usage guidance.

llama.cpp setup

Build the G9v3-enabled branch:

git clone --branch feature/g9v3-support --single-branch \
  https://github.com/linuxid10t/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build --target llama-cli llama-server -j

Command line

Both GGUF files contain the model's chat template. Start an interactive

conversation with the Q4_K_M file using:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -cnv

For BF16, use the same command with the other filename:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-BF16.gguf \
  -c 4096 -t 32 -cnv

Start with a modest context such as 4,096 tokens, then increase it according to

your available memory and workload.

OpenAI-compatible server

./llama.cpp/build/bin/llama-server \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -tb 32 \
  --host 127.0.0.1 --port 8080

Replace the model filename with G9v3-39A5B-BF16.gguf to serve the BF16

version on a machine with sufficient memory.

Example request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "G9v3-39A5B-Q4_K_M.gguf",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 256
  }'

Bind the server to a non-loopback address only after configuring appropriate

authentication and network controls.

Conversion provenance

The source BF16 checkpoint was converted to GGUF and then quantized with:

python convert_hf_to_gguf.py G9v3-39A5B \
  --outfile G9v3-39A5B-BF16.gguf \
  --outtype bf16

./build/bin/llama-quantize \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-Q4_K_M.gguf \
  Q4_K_M 32

Conversion environment:

• llama.cpp base commit: 9d57ce456 (b10436)
• G9v3 support commit: 77b13da1a
• GGUF version: V3
• Tensor count: 568 in each file
• BF16 weight size: 74,371.15 MiB, 16.01 bits per weight
• Q4_K_M weight size: 22,462.90 MiB, 4.84 bits per weight

Validation and performance

The Q4_K_M model was tested end-to-end with both llama-cli and

llama-server on a CPU-only system. The performance figures below apply only

to Q4_K_M.

| Hardware | Context | Threads | Prompt processing | Generation |

| --- | ---: | ---: | ---: | ---: |

| AMD Ryzen 9 9950X, 64 GiB RAM, CPU only | 4,096 | 32 | 50.18–57.9 tokens/s | 10.0–10.03 tokens/s |

The exact rate will vary with prompt length, thread count, memory speed, build

options, and operating system.

Validation performed:

• Successful load and text generation from the Q4_K_M file
• Successful /v1/chat/completions request with separate reasoning content
• BF16 GGUF metadata and all 568 tensor shapes loaded successfully
• BF16 file successfully used as the source for Q4_K_M quantization
• Converter dry run passed against all 15 source checkpoint shards after

rebasing onto llama.cpp b10436

• Synthetic G9v3 CPU architecture test: zero numerical error
• GGUF serialization round-trip test: passed

The BF16 file was not generation-benchmarked on the 64 GiB validation machine;

it requires a system with more memory for representative testing.

Sampling

The base model authors recommend the following settings:

| Mode | Temperature | Top-p |

| --- | ---: | ---: |

| Think | 1.0 | 0.95 |

| No Think | 0.7 | 0.95 |

Limitations

• Q4_K_M is a lossy quantization and may perform differently from the BF16

file and original checkpoint.

• The BF16 GGUF preserves source precision but has much higher memory and

storage requirements.

• G9v3 is a preview model and may produce inaccurate, biased, or unsafe output.
• Outputs should be independently reviewed before use in medical, legal,

financial, safety-critical, or other high-stakes settings.

• This GGUF currently depends on the linked G9v3-enabled llama.cpp branch.
• Long-context capability was not benchmarked as part of this conversion test.

License and attribution

The original model is released by AI9Stars under the Apache License 2.0. This

conversion retains that license. See the base model repository for its complete

license terms and attribution information.

This conversion is independently produced and is not an official AI9Stars

release.