---

license: apache-2.0

language:

- en

base_model: Qwen/Qwen3-Next-80B-A3B-Instruct

base_model_relation: finetune

pipeline_tag: text-generation

tags:

- gguf

- qwen3_next

- mixture-of-experts

- llama.cpp

- conversational

- tool-calling

- graft

- ultralabs

---

GRAFT-50B-A3B

GRAFT-50B-A3B is a ~50B-parameter mixture-of-experts chat model from UltraLabs. Only about

3B parameters are active per token, so it generates text at roughly the speed of a 3B model while carrying

the knowledge of a much bigger one.

It was made by shrinking Qwen3-Next-80B-A3B-Instruct

to under two thirds of its size with GRAFT, our own model-shrinking method, then healing it by distillation

from the original 80B model and teaching it its own name.

• Made by: UltraLabs (SmallAICreator)
• Format: GGUF, GRAFT-50B-A3B-Q4_K_M.gguf (30.9 GB)
• Size: 50.7B parameters total, 3.2B active per token
• Architecture: Qwen3-Next hybrid, 48 layers (36 Gated DeltaNet + 12 gated attention), 320 experts

(10 active per token + 1 shared expert), hidden size 2048

• Chat template: ChatML with tool calling (embedded in the GGUF)
• Knows who it is: no system prompt needed
• Language: English first
• License: Apache 2.0

---

Quick start

Needs a recent llama.cpp (Qwen3-Next support).

llama.cpp

# Chat in the terminal
llama-cli -hf SmallAICreator/GRAFT-50B-A3B-GGUF:Q4_K_M -cnv \
  --temp 0.7 --top-p 0.8 --top-k 20 --repeat-penalty 1.05

# OpenAI-compatible server on http://localhost:8080 (32K context)
llama-server -hf SmallAICreator/GRAFT-50B-A3B-GGUF:Q4_K_M --jinja -c 32768 \
  --temp 0.7 --top-p 0.8 --top-k 20 --repeat-penalty 1.05

On a GPU with less VRAM than the model, keep some experts in system RAM with --n-cpu-moe N

(the number of layers whose experts stay on the CPU) and lower N until your VRAM is full.

Apps

Any GGUF app with Qwen3-Next support works. Download GRAFT-50B-A3B-Q4_K_M.gguf and load it; the chat template

is read from the file.

Hardware

| | |

| --- | --- |

| Memory | ~31 GB for the model: 32 GB RAM + a GPU, or 48 GB+ of RAM / VRAM |

| CPU only | works if it fits in RAM; 8.8 tokens/s measured on a 24-core server CPU |

| GPU | much faster; a single 48 GB GPU holds it completely |

| Long context is cheap | only 12 of the 48 layers keep a KV cache, so 32K of context needs well under 1 GB |

---

Recommended settings

| Setting | Value |

| --- | --- |

| Temperature | 0.7 |

| Top-p | 0.8 |

| Top-k | 20 |

| Repeat penalty | 1.05 |

Prompt format

ChatML. Tool calls use the <tool_call> format, and the embedded template handles OpenAI-style tools.

A system prompt is optional: the model already knows its name.

<|im_start|>user
Who are you?<|im_end|>
<|im_start|>assistant
I'm GRAFT-50B-A3B, an AI assistant made by UltraLabs. I can answer questions, write and explain code, help with math, and chat.<|im_end|>

---

Sample replies

Greedy outputs with no system prompt, from the final model:

> Who are you?

>

> I'm GRAFT-50B-A3B, an AI assistant made by UltraLabs. I can answer questions, write and explain code, help with

> math, and chat.

> Who made you?

>

> I was made by UltraLabs.

> hey, what's your name?

>

> My name is GRAFT-50B-A3B.

> What is the capital of France?

>

> The capital of France is Paris.

It only introduces itself when asked: on ordinary questions (facts, code, math, advice) it just answers.

Benchmarks

All run on this exact Q4_K_M file with llama.cpp, greedy decoding, no system prompt:

| Benchmark | Setup | GRAFT-50B-A3B |

| --- | --- | --- |

| MMLU (full test set) | 14,042 questions, zero-shot, answer letter | 77.4% |

| GSM8K | first 250 test problems, step by step, final number | 97.2% |

| HumanEval | all 164 problems, pass@1, code run against the tests | 84.1% |

| Needle in a haystack | 5-digit passkey at 10% / 50% / 90% depth | 9/9 (3/3 at 8K, 16K and 32K) |

How much the healing recovered. On a fixed 285-question MMLU subset (5 per subject, same prompts and scoring):

| Model | MMLU subset |

| --- | --- |

| Qwen3-Next-80B-A3B-Instruct (the original) | 85.6% |

| GRAFT-50B-A3B before healing | 76.1% |

| GRAFT-50B-A3B (this release) | 78.9% |

Strongest MMLU areas: history, government and politics, psychology, foreign policy and marketing (92-98%). Weakest:

college-level chemistry, physics and math (50-58%).

These are our own zero-shot runs (not the official 5-shot setups), so compare them with other models run the same

way rather than with leaderboard numbers.

Limitations

• College-level science and math are its weakest knowledge areas (about 50-58% on those MMLU subjects).

Grade-school math word problems are strong (97% on GSM8K). Check important calculations.

• It talks confidently. Check important facts, and run and test generated code.
• Tuned in English. Other languages work but are less polished.

Credits

• Base model: Qwen3-Next-80B-A3B-Instruct by the Qwen

team (Apache 2.0).

• Shrinking, healing and identity tuning: UltraLabs (SmallAICreator).
• Runs on llama.cpp.

Also from UltraLabs: GRAFT-10B-A2B,

GRAFT-1B and the

L-AI Android app for running GGUF models locally.