cafepm/G9v3-39A5B-Q3Q4-GGUF overview

---

license: apache-2.0

language:

• en
• zh

library_name: transformers

pipeline_tag: text-generation

tags:

• g9v3
• moe
• text-generation
• long-context
• tool-calling
• preview
• RSIT
• Infiniteons
• Q3Q4
• TNT

---

<center><a href="https://ai.tnt.chat" target="_blank">

<img src="https://ai.tnt.chat/assets/tnt.chat.shape-white.bg-transp-Dj8x3xII.svg" width="30%">

</a></center>

<img src="https://ai.tnt.chat/article/g9v3-artificial_analysis.jpg" width="100%">

We tested G9v3, and it is the strongest model under 60B parameters in our tnt-bench. Its first-place result in the NON-HALLUCINATION benchmark is striking, and we had to create a proprietary fork of llama.cpp to run the model in quantized form. The model is not particularly quantization-friendly. Unlike Muse Glimmer, which handled Q2 extremely well, G9v3 needed Q3 to achieve its full potential. Even so, it is still small enough to run on a single 24 GB VRAM GPU.

Quantized from ai9stars/G9v3-39A5B using the Infiniteon Algebra associator to guide mixed-precision allocation: 20% of the weights—the shared expert and attention tensors—are quantized to 4-bit (Q4_K), while the remaining 80% are quantized to 3-bit (Q3_K). This achieves Q4-level generation quality at near-Q3 size, allowing a 39B-parameter model to fit within 24–32 GB of VRAM.

Why this works: The Infiniteon Algebra associator identifies which weight tensors carry structural "gap-crossing" information—the tensors that connect diverse token neighborhoods. By allocating 4-bit precision to these structurally critical tensors and 3-bit precision to the rest, the model preserves the directional diversity required for coherent generation while keeping the overall size close to Q3.

Research and credits: This research was conducted by TNT.Chat (https://ai.tnt.chat) as part of our LOCAL AI effort to bring larger models to consumer hardware. The goal is to enable capable models to run on consumer GPUs such as the RTX 5090 for agentic workloads with large context windows.

Theory references:

Pinto Martins, C. F. (2026). Ramsey Statistics And Infiniteons Theory, Volume I. Zenodo. DOI: https://doi.org/10.5281/zenodo.19329589

Pinto Martins, C. F. (2026). Ramsey Statistics And Infiniteons Theory, Volume II. Zenodo. DOI: https://doi.org/10.5281/zenodo.19330373

<img src="https://ai.tnt.chat/article/g9v3-comparison.jpg" width="90%">

Our overall pick for running a large context window on a 24 GB GPU is the Q3Q4 format.

For more details about tnt-bench, please visit our website.

---

<div align="center">

<h1>G9v3-39A5B</h1>

<p><b>⚠️ Preview Release</b></p>

</div>

<p align="center">

<a href="https://github.com/AI9Stars" target="_blank">GitHub</a> |

<a href="https://huggingface.co/ai9stars" target="_blank">Hugging Face</a>

</p>

> This is a preview release. G9v3-39A5B is under active development — expect continued updates with improved performance and additional capabilities in the near future.

Introduction

G9v3-39A5B is a Mixture-of-Experts (MoE) causal language model from the AI9Stars team, with 39B total parameters and 5B activated per token. The sparse design keeps inference cost close to a small dense model while retaining the capacity of a much larger one, making it a practical choice for local and self-hosted deployment.

It targets everyday assistant use, coding, tool-use workflows, and reasoning tasks, and supports both Think / No Think modes through the same checkpoint.

Model Information

• Type: Causal Language Model (Mixture-of-Experts)
• Total Parameters: ~39B
• Activated Parameters: ~5B per token
• Context Length: 131,072

Quickstart

vLLM

pip install "vllm>=0.21"
vllm serve ai9stars/G9v3-39A5B --port 8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai9stars/G9v3-39A5B",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

SGLang

pip install "sglang[srt]>=0.5.12"
python -m sglang.launch_server --model-path ai9stars/G9v3-39A5B --port 30000

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai9stars/G9v3-39A5B",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

Transformers

pip install -U "transformers>=5.6" accelerate torch

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ai9stars/G9v3-39A5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [{"role": "user", "content": "Who are you?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Recommended sampling parameters:

| Mode | Recommended params | Enable |

| --- | --- | --- |

| Think | temperature=1.0, top_p=0.95 | enable_thinking=True |

| No Think | temperature=0.7, top_p=0.95 | enable_thinking=False |

Limitations and Responsible Use

G9v3-39A5B is a language model that generates content based on learned statistical patterns from training data. It may produce inaccurate, biased, or unsafe outputs, and generated content should be reviewed and verified before use in high-stakes settings. As a preview release, its behavior may change between versions. Users are responsible for evaluating outputs, applying appropriate safeguards, and complying with applicable laws, regulations, and platform policies.

License

This repository and the G9v3 model weights are released under the Apache-2.0 License.