---

license: apache-2.0

base_model:

• ai9stars/G9v3-39A5B

tags:

• gguf
• moe
• text-generation

---

G9v3-39A5B — Q2 GGUF

Fast GGUF for mainstream llama.cpp. Requires a build with G9v3 architecture

support (e.g. linuxid10t/llama.cpp@feature/g9v3-support) until upstream

llama.cpp merges it — standard builds will not load this model.

Recipe (antirez style)

| Tensor group | Format |

| --- | --- |

| Routed expert gate / up | IQ2\_XXS (imatrix) |

| Routed expert down | Q2\_K |

| Shared experts, attention, output | Q8\_0 |

| Token embeddings | F16 |

| Routers, norms, biases | F32 |

Same strategy as Salvatore Sanfilippo's

DeepSeek V4 quants:

imatrix-calibrated IQ2_XXS gate/up with K-quant down routed experts, Q8_0

protection everywhere else. Imatrix:

linuxid10t.

Files

| File | Size |

| --- | ---: |

| G9v3-39A5B-Q2-IQ2_XXS-Q2_K.gguf | 11.21 GiB |

Validation

gguf-dump audit: all routed gate/up are IQ2_XXS, all down are Q2_K

(native 256-wide rows, no padding needed), zero misquantized experts;

attn/shexp/output Q8_0, token F16, norms/router F32. Spot ppl + KL

divergence vs the Q8_0 reference on short prompts (arithmetic, Italian

fact, C completion, counting, colors), CPU-only: Q8 PPL 5.21, Q2 PPL 6.77,

KL 0.21, same top token 83.9%.

Usage

# needs a G9v3-enabled llama.cpp build, e.g.
# git clone -b feature/g9v3-support https://github.com/linuxid10t/llama.cpp
llama-cli -m G9v3-39A5B-Q2-IQ2_XXS-Q2_K.gguf -t 16 -c 32768

# server
llama-server -m G9v3-39A5B-Q2-IQ2_XXS-Q2_K.gguf -t 16 -c 32768 --host 0.0.0.0 --port 8080

Model

G9v3-39A5B is a 320-expert MoE preview model by AI9Stars. Context: long.

Original model: ai9stars/G9v3-39A5B.

Original BF16 GGUF: linuxid10t/G9v3-39A5B-GGUF.

License: Apache-2.0.