---

license: apache-2.0

base_model: logic65/Whittle-Qwen-3.8-35B-A3B

base_model_relation: quantized

quantized_by: logic65

library_name: llama.cpp

pipeline_tag: text-generation

language:

- en

- zh

- multilingual

tags:

- gguf

- llama.cpp

- whittle

- qwen4_exp

- mixture-of-experts

- moe

- n-gram-memory

- conditional-memory

- knowledge-distillation

- reasoning

- thinking

- code

- math

- text-generation

- conversational

- research

---

<p align="center"><img src="https://huggingface.co/logic65/Whittle-Next-27B-A3B/resolve/main/whittle.svg" width="640" alt="Whittle"></p>

Whittle-Qwen-3.8-35B-A3B — GGUF

> ### ☕ Support this work

> Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished:

> ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

Ready-to-run llama.cpp quantisations of logic65/Whittle-Qwen-3.8-35B-A3B: a 35.1 B-total, ~3 B-active mixture-of-experts in the

Qwen3.8-Flash-Next (qwen4_exp) format with a 10 B-parameter n-gram memory that is load-bearing (the body measurably needs it; see the base card).

Distilled from Qwen3.8-27B at the logit level on 25k verified teacher traces; built on Whittle-Next-27B-A3B. Runs on stock llama.cpp, no patches. The full card, measurements and caveats

are on the base repo; this repo is the download.

Status

Current. These files are the Phase-2 step 32010 root (p2-s32010, 30 Sep 2026) of the current Whittle flagship: the complete Phase-2 distillation from Qwen3.8-27B (every usable teacher row seen once), a memory

phase, and a 410-step fix for the llama.cpp web page's browser tools. The previous (p2-s6000) ladder was replaced in place; the earlier full weights stay under bf16-s6000/, bf16-agentfix2/, bf16-lw5/,

bf16-lw2/ and bf16-tbl1/ on the base repo.

Which file

The memory is ~10.5 GB of each file at Q8 (proportionally less at lower bits). It is a lookup, one row per token per head, so keep it in system

RAM with -ot per_layer_token_embd=CPU — the GPU then holds a 27 B-class footprint and generation runs at 3 B-active speed.

| file | size | notes |

|---|---|---|

| Whittle-Qwen-3.8-35B-A3B-Q8_0.gguf | 37.8 GB | reference quant. The base card's numbers were measured on a Colab conversion of the same checkpoint; this file (converted on our 3× RTX 3060 box) was spot-checked against them: web-page chat replay 19/20 and 16/20, 0 invented tools, bug-report JSON 12/12 (the base card's grader) |

| Whittle-Qwen-3.8-35B-A3B-Q6_K.gguf | 29.3 GB | near-lossless |

| Whittle-Qwen-3.8-35B-A3B-Q5_K_M.gguf | 25.1 GB | recommended for one 24 GB card with the memory in RAM |

| Whittle-Qwen-3.8-35B-A3B-Q4_K_M.gguf | 21.3 GB | good default for 16–20 GB cards, memory in RAM |

| Whittle-Qwen-3.8-35B-A3B-Q3_K_M.gguf | 16.7 GB | smallest; expect some loss on maths |

K-quants were requantised from the Q8_0. Every file carries the updated chat template, which accepts a system message anywhere in the conversation, so Claude Code works (it sends its environment block after the user turn).

Serve the table whole: the body depends on the memory (see the base card); a build that drops or re-hashes

per_layer_token_embd behaves like the v4.4 parent minus its knowledge.

Measured (summary; the full tables are on the base card)

Served as the Q8_0 it scores 23/24 on the stop/loop battery, 43/50 on unseen GSM8K, 46/60 on our MATH probe,

72/72 exact on a strict-JSON hold-out and 96/96 on our tool-use probes, and it finishes 31/36 held-out chats on the llama.cpp web page with its browser tools on (the step

before the fix: 15/36); see the base card.

Run it

llama-server -m Whittle-Qwen-3.8-35B-A3B-Q5_K_M.gguf -ngl 99 -c 16384 --jinja -fa on -ot per_layer_token_embd=CPU

• sampler: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05 — sample, do not decode greedily; greedy decoding loops on this family.
• thinking: "chat_template_kwargs": {"enable_thinking": true} — it does its best work with thinking on (distilled on complete thinking traces).

It thinks longer than earlier checkpoints: give it max_tokens 4096+ for code and 8192+ for maths. --reasoning-format deepseek separates the thinking into reasoning_content.

• To also move the routed experts to RAM on small cards: -ot "per_layer_token_embd=CPU" -ot "\.ffn_(up|down|gate)_exps\.=CPU".
• On llama.cpp builds that have --lazy-mode (newer than ~30 Aug 2026), add --lazy-mode off so the memory table is loaded into RAM; otherwise it is read lazily from disk and -ot is ignored, which makes prompt processing slow.
• Tool use / agents: keep each assistant turn's reasoning_content in the history for the rest of the tool episode.
• Architecture qwen4exp; if your build reports an unknown architecture, update llama.cpp.

Provenance

David Aylward (logic65) & Claude (Anthropic). Parent: logic65/Whittle-Next-27B-A3B. Teacher: Qwen/Qwen3.8-27B.

Memory contents: Qwen/Qwen3.8-Flash-Next. All Apache-2.0.