---

license: apache-2.0

license_link: https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B/blob/main/LICENSE

base_model:

- Qwen/Qwen-AgentWorld-35B-A3B

base_model_relation: quantized

language:

- en

pipeline_tag: text-generation

library_name: gguf

quantized_by: Frosty40

tags:

- gguf

- llama.cpp

- qwen35moe

- mixture-of-experts

- agent

- world-model

- environment-simulation

- intel-arc

- sycl

- b70

---

Qwen-AgentWorld-35B-A3B — B70 Turbo (GGUF)

GGUF builds of Qwen/Qwen-AgentWorld-35B-A3B

tuned and validated for fast local agent inference on the Intel Arc Pro B70 with llama.cpp (SYCL).

This repository ships the Q5_K_M agent-augmented quant — the best quality/throughput all-rounder

from an extended B70 optimization campaign — together with the exact serving recipe, full throughput and

quality benchmarks, and an optional speculative-decode route for deep structured contexts.

> TL;DR — Q5_K_M-agentaug, 34.7B total / ~3B active MoE, 5.71 bpw, 24.7 GB. On a single Arc Pro B70

> it sustains ~85 tok/s decode on short agent/tool turns and >1,200 tok/s prefill, holding **~16.5

> tok/s decode at 73k context — and 150–224 tok/s aggregate across a 32-agent fleet** (2.4–3.5× via

> concurrency). Quant is near-lossless to the BF16 source (see Quality). Apache-2.0.

---

One Shot Game Prompts: https://agentic-arcade.vercel.app/model-agentworld.html

Where the speed comes from

Three configs, same Q5_K_M weights, same f16 KV, same GPU:

1. upstream — mainline llama.cpp, default flags
2. up+flags — same mainline + Turbo flags (GGML_SYCL_DISABLE_DNN=1 -b 8192 -ub 4096)
3. Turbo (this build) — same mainline + 3 fused-kernel commits (topk-MoE router fusion,

gate-glue fusion, single-token expert-aggregate) + same flags

Summary: the tuned runtime configuration (-b 8192 -ub 4096, oneDNN GEMM disabled) delivers the

prefill (1.2–1.7×) and fleet (~1.3–1.4×) gains and applies to any llama.cpp SYCL build. The Turbo

build's fused decode kernels add a further +7–14% to single-stream decode. Output quality is

unchanged — identical weights, lossless serving.

• Prefill 1.2–1.7× and fleet ~1.3–1.4× come from the tuned runtime configuration, reproducible

on stock upstream llama.cpp (the Turbo build contributes 1.00–1.01× on top).

• Single-stream decode +7–14% at every depth (0.8k→129k) is delivered by the Turbo build's fused

kernels.

• Trade-offs: at 8–16 concurrent agents, fused decode performs within ~2% of the unfused path

(build win 0.98×; neutral at 24+). -ub 4096 trades VRAM (larger compute buffer) for the prefill win.

• Quality: identical weights, lossless.
• Provenance: the MoE reorder is merged upstream (ggml-org/llama.cpp PR #24452); the fused

kernels above are specific to this build.

Full breakdown: see "Performance breakdown" under Throughput, below.

---

Model details

| | |

|---|---|

| Base model | Qwen/Qwen-AgentWorld-35B-A3B (agent / environment world-model on Qwen3.5-35B-A3B-Base) |

| Architecture | qwen35moe — MoE + gated-delta-net (linear-attn) layers |

| Layers | 40 |

| Experts | 256 routed (8 active) + 1 shared, expert FFN 512 |

| Hidden size | 2048, head_dim 256 (16 Q heads / 2 KV heads, GQA) |

| Linear-attn | gated-delta-net SSM (state 128, conv 4, groups 16), full-attention every 4th layer |

| Total params | 34.66 B |

| Active params / token | ~3 B (A3B) |

| Native context | 262,144 (tested/served at 131,072) |

| RoPE θ | 1.0e7 |

| Vocab | 248,320 (BPE / qwen35 pre-tokenizer) |

| This quant | Q5_K_M (imatrix, agent-augmented calibration) |

| Tensor mix | Q5_K ×371, Q6_K ×61, F32 ×301 (733 tensors) |

| Effective bpw | 5.708 bits/param |

| File size | 24.73 GB (23.03 GiB) |

| License | Apache-2.0 (inherited from base) |

SHA256 (agentworld-35b-a3b-Q5_K_M-agentaug.gguf):

27a4c510844330e4980c96bd0d80f1511bacc9557aa7b21f812a81eeebd5d7b4

---

Files

| File | Quant | Size | bpw | Notes |

|---|---|---:|---:|---|

| agentworld-35b-a3b-Q5_K_M-agentaug.gguf | Q5_K_M | 24.7 GB | 5.71 | Recommended. Agent-augmented imatrix. Fits the B70's 30.3 GiB with full 131k KV. |

> Other quants (Q4_K_M speed / Q6_K quality / Q8_0 near-lossless reference) were produced during the

> campaign and can be added on request.

---

Throughput on the Intel Arc Pro B70

Measured end-to-end through the llama.cpp OpenAI-compatible server (/v1/chat/completions, real agent

prompts), single B70, ship config below. temperature=0, 512 generated tokens per request.

Decode & prefill vs. context depth (averaged over the two well-calibrated prompt classes,

structured tool/log/code and novel long-context synthesis):

| Context depth | Decode tok/s | Prefill tok/s |

|---:|---:|---:|

| short turn (~0.1–0.7k) | 84.4 – 85.8 | 246 – 695 |

| ~24k | 35.9 | ~1,240 |

| ~45k | 23.8 | ~950 |

| ~73k | 16.5 | ~715 |

Short agent/tool turns (decode-dominated, the common interactive case):

| Prompt | prompt_n | Prefill tok/s | Decode tok/s |

|---|---:|---:|---:|

| shell-log triage | 125 | 292.3 | 85.8 |

| weather tool reply | 95 | 245.8 | 85.7 |

| checkout-DOM next-action | 94 | 257.2 | 85.0 |

Full per-request matrix (4 classes × depth buckets): throughput_default.tsv.

Prefill peaks above 1,200 tok/s around 24k context; decode degrades gracefully with depth (KV-bandwidth

bound), staying usable (~16 tok/s) at 73k. Decode is memory-bandwidth bound on this MoE, so smaller quants

do not decode meaningfully faster — Q5_K_M is the quality/throughput sweet spot.

Canonical llama-bench

Reproducible pp512 / tg128 at matched KV depths (ship config: -fa on, q8_0 KV, -r 3,

build 0b98ca1b4):

| Test | tok/s |

|---|---:|

| pp512 | 1190.9 ± 12.0 |

| tg128 | 87.2 ± 0.2 |

| pp512 @ d24000 | 703.3 ± 8.6 |

| tg128 @ d24000 | 36.5 ± 0.0 |

| pp512 @ d45000 | 508.7 ± 2.0 |

| tg128 @ d45000 | 24.2 ± 0.0 |

| pp512 @ d73000 | 378.8 ± 1.1 |

| tg128 @ d73000 | 16.7 ± 0.0 |

(Pure-generation llama-bench figures run ~2% above the end-to-end server numbers above, which include

chat-template + HTTP overhead — the two cross-validate cleanly.)

Concurrent serving (agent fleets)

One decode stream under-feeds the B70 (it is memory-bandwidth bound), so running **parallel agents

multiplies aggregate throughput**. Serving with -np N parallel slots, realistic agent turns:

| Parallel agents (-np) | Aggregate decode tok/s | Per-agent decode tok/s |

|---:|---:|---:|

| 1 | 76 | 76 |

| 8 | 108 | 13.5 |

| 16 | 128 | 8.0 |

| 32 (recommended) | 160 – 224 | 5 – 8 |

| 48 (peak) | 178 | 3.7 |

A single B70 sustains 150–224 tok/s aggregate across 32 concurrent agents (real short tool turns hit

223 tok/s) — a 2.4–3.5× total-throughput gain over one agent, ideal for autonomous/background agent

fleets. -np 32 is the recommended production point (knee of the curve, comfortable VRAM headroom); `-np

48 is the clean peak. Do not exceed -np 48 — at -np 56` the 30 GiB is exceeded and throughput falls

off a cliff (≈20× collapse). Concurrency trades against per-agent context depth (agents × context must fit

30 GiB), so reserve high -np for shallow, high-volume agent traffic and serve deep-context work at `-np

1–2. See BENCHMARKS.md` §6.

Performance breakdown — flags vs. build (AgentWorld-35B-A3B, f16 KV)

The figures above combine two independent effects: runtime flags (`GGML_SYCL_DISABLE_DNN=1 -b 8192

-ub 4096`) and this build's fusion commits. Isolated on the same Q5_K_M weights, same GPU:

flags win = (2)/(1) (mainline llama.cpp, same flags) · build win = (3)/(2) (Turbo build, same flags).

Prefill t/s vs prompt tokens

| prompt | upstream | up+flags | Turbo | flags | build | total |

|--:|--:|--:|--:|:--:|:--:|:--:|

| 805 | 1099 | 1397 | 1405 | 1.27× | 1.01× | 1.28× |

| 3313 | 1096 | 1845 | 1850 | 1.68× | 1.00× | 1.69× |

| 6963 | 1058 | 1736 | 1738 | 1.64× | 1.00× | 1.64× |

| 14563 | 976 | 1535 | 1537 | 1.57× | 1.00× | 1.58× |

| 29713 | 835 | 1205 | 1208 | 1.44× | 1.00× | 1.45× |

| 61341 | 634 | 827 | 827 | 1.30× | 1.00× | 1.30× |

| 129325 | 414 | 488 | 488 | 1.18× | 1.00× | 1.18× |

Decode t/s vs context depth

| depth | upstream | up+flags | Turbo | flags | build | total |

|--:|--:|--:|--:|:--:|:--:|:--:|

| 805 | 81.7 | 81.7 | 93.6 | 1.00× | 1.14× | 1.15× |

| 3313 | 80.0 | 79.8 | 91.3 | 1.00× | 1.14× | 1.14× |

| 6963 | 77.5 | 77.4 | 88.3 | 1.00× | 1.14× | 1.14× |

| 14563 | 73.0 | 72.7 | 82.3 | 1.00× | 1.13× | 1.13× |

| 29713 | 66.6 | 66.1 | 74.1 | 0.99× | 1.12× | 1.11× |

| 61341 | 55.7 | 55.5 | 61.0 | 1.00× | 1.10× | 1.10× |

| 129325 | 41.5 | 41.3 | 44.3 | 1.00× | 1.07× | 1.07× |

Fleet aggregate decode t/s vs agents (2048+256 synthetic)

| agents | upstream | up+flags | Turbo | flags | build | total |

|--:|--:|--:|--:|:--:|:--:|:--:|

| 1 | 78.5 | 79.0 | 86.1 | 1.01× | 1.09× | 1.10× |

| 2 | 73.2 | 72.4 | 73.9 | 0.99× | 1.02× | 1.01× |

| 4 | 93.3 | 119.0 | 120.9 | 1.28× | 1.02× | 1.30× |

| 8 | 102.3 | 142.0 | 138.7 | 1.39× | 0.98× | 1.36× |

| 16 | 107.5 | 146.2 | 144.0 | 1.36× | 0.98× | 1.34× |

| 24 | 117.6 | 158.2 | 158.2 | 1.35× | 1.00× | 1.35× |

| 32 | 125.1 | 168.0 | 166.7 | 1.34× | 0.99× | 1.33× |

| 40 | 132.4 | 172.8 | 172.0 | 1.31× | 1.00× | 1.30× |

| 48 | 138.2 | 177.6 | 177.6 | 1.29× | 1.00× | 1.29× |

| 56 | 142.2 | 181.4 | 180.9 | 1.28× | 1.00× | 1.27× |

Summary: the tuned runtime configuration (-b 8192 -ub 4096, oneDNN GEMM disabled) delivers the

prefill (1.2–1.7×) and fleet (~1.3–1.4×) gains and applies to any llama.cpp SYCL build. The Turbo

build's fused decode kernels add a further +7–14% to single-stream decode at every depth. At 8–16

concurrent agents, fused decode performs within ~2% of the unfused path (build win 0.98×; neutral at

24+). Output quality is unchanged — identical weights, lossless serving. Measured on plain Q5_K_M;

the shipped agentaug imatrix variant has the same quant type and tensor layout, so throughput is

identical. Raw data: AgentWorld-35B-A3B-B70-Turbo.

---

Quality

Quantization fidelity vs. the high-precision source, plus held-out perplexity on agent-domain and generic

text. PPL measured with llama-perplexity (-ngl 99, fp16 KV, 512-token windows).

Perplexity (lower = better; held-out sets, ~23k tokens each):

| Eval set | Q5_K_M agent-aug (shipped) | Q5_K_M base imatrix |

|---|---:|---:|

| Agent-domain held-out | 4.0601 ± 0.080 | 4.0772 ± 0.081 |

| Generic held-out | 7.8559 ± 0.198 | 7.8566 ± 0.198 |

The agent-augmented imatrix lowers agent-domain PPL by −0.42% with no regression on generic text

(−0.009%) — the calibration choice is domain-appropriate and free.

KL-divergence vs. Q8_0 reference — how faithfully Q5_K_M reproduces the higher-precision model's

output distribution (agent-domain text):

| Metric | Q5_K_M agent-aug (shipped) |

|---|---:|

| Mean KLD | 0.00639 ± 0.0005 |

| Median KLD | 0.00050 |

| 95th percentile KLD | 0.0268 |

| 99th percentile KLD | 0.0794 |

| Same top-1 token as Q8 | 97.35% |

| PPL(Q5) / PPL(Q8) | 1.0021 |

Q5_K_M tracks the Q8_0 reference very closely — mean KLD ~0.006, top-token agreement ~97%, and

perplexity within 0.2%. (Measured over a time-boxed 4k-token agent-domain sample; the base-imatrix variant

is statistically equivalent at Mean KLD 0.00671 / 97.45% same-top.) For reference-grade fidelity, the

Q8_0 build is available on request.

The shipped quant uses an agent-augmented importance matrix: the base calibration corpus (generic) was

blended ~31% with authored agent-domain text (tool-call XML, terminal/k8s/JSON/log/code/reasoning traces)

so the Q5_K_M rounding preserves the tensors that matter most for agentic use. The effect is small but in

the right direction and domain-appropriate (see PPL table; agent text improves, generic does not regress).

---

How to run (Intel Arc B70, llama.cpp SYCL)

Built from llama.cpp with the SYCL backend (oneAPI). The validated default-turbo route:

GGML_SYCL_DISABLE_DNN=1 \
  llama-server \
  -m agentworld-35b-a3b-Q5_K_M-agentaug.gguf \
  --alias agentworld-35b-a3b-turbo \
  -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 131072 -np 1 -b 8192 -ub 4096 \
  --host 127.0.0.1 --port 8091 --jinja

Why these flags:

• GGML_SYCL_DISABLE_DNN=1 — native SYCL GEMM ≥ oneDNN for this MoE shape, and gives deterministic decode.
• -ngl 99 — all 40 layers + experts fit the B70's 30.3 GiB at Q5_K_M with full 131k KV.
• -fa on -ctk q8_0 -ctv q8_0 — flash-attention + q8_0 KV cache: large context within VRAM, q8_0 chosen

over q4_0 (q4_0 was both slower and lower quality on this shape).

• -b 8192 -ub 4096 — the prefill sweet spot (a large micro-batch is the dominant prefill win here).

Then call it as an OpenAI-compatible endpoint (/v1/chat/completions). It is a thinking model — it

emits a <think> reasoning block before the answer.

Optional: structured-deep speculative route

For very large structured contexts (logs / code / tool output / observation echo) an n-gram speculative

decoder gives a real speed-up at depth (it drafts from the repetitive structure). On the same model:

... llama-server ... \
  --spec-type ngram-map-k4v \
  --spec-ngram-map-k4v-size-n 8 --spec-ngram-map-k4v-size-m 24 \
  --spec-ngram-map-k4v-min-hits 1 --spec-draft-n-max 24

Measured +19% decode at 45k and +37% at 73k on structured incident-analysis prompts (byte-identical

output to the non-spec baseline at 73k). It is opt-in / route-by-class: it does not help novel prose

and is roughly neutral at shallow depth, so route only structured-deep traffic to it.

Recommended sampling

The base model's defaults (carried in the GGUF): temperature 0.6, top_p 0.95, top_k 20.

Use temperature 0 for deterministic agent/eval runs.

---

Example outputs

Real generations from this quant on the B70 (temperature 0), illustrating the two core modes.

Tool-result → user-facing reply (decode-dominated agent turn, 83 tok/s):

> User: get_weather returned {"location":"Austin, TX","temp_f":98,"humidity":0.41,"wind_mph":12,"conditions":"clear"}. Write the assistant reply.

>

> AgentWorld: The current weather in Austin, TX is clear with a temperature of 98°F. The humidity is at

> 41%, and the wind is blowing at 12 mph.

Structured incident triage → strict JSON (the structured-deep use case):

> User: (kubectl pod CrashLoopBackOff + `ERROR migration check failed code=E_SCHEMA_MISSING

> table=ledger_events + /healthz {"ready":false,"db":"ok","schema":"missing"}`) — return JSON with

> root_cause, evidence, next_command.

>

> AgentWorld:

> ```json

> {"root_cause": "Database schema migration failed due to missing 'ledger_events' table, causing the

> payments-api pod to enter CrashLoopBackOff.",

> "evidence": "kubectl.logs shows 'ERROR migration check failed code=E_SCHEMA_MISSING

> table=ledger_events', and http.get /healthz returns {'ready':false,'db':'ok','schema':'missing'}.",

> "next_command": "Apply database migrations to create the missing 'ledger_events' table schema."}

> ```

It reasons inside a <think> block first, then emits the final answer (clean prose or strict JSON as asked).

---

Provenance & reproducibility

• Source: Qwen/Qwen-AgentWorld-35B-A3B → convert_hf_to_gguf.py --no-mtp → BF16 GGUF → imatrix →

Q5_K_M. The published checkpoint is language-model-only (no MTP/vision weights shipped), so it is

converted with --no-mtp.

• Importance matrix: agent-augmented calibration (~31% authored agent/tool/world-model text blended

into a wide generic base corpus).

• Engine: llama.cpp SYCL build for Battlemage (the Turbo build). The MoE mul_mat_id

reorder is merged upstream (ggml-org/llama.cpp PR #24452); top-k-MoE router fusion + gated-delta-net

gate fusion + expert-combine fusion are specific to this build (3 commits) and drive the

single-stream decode gain — prefill/fleet throughput comes from the runtime flags alone, reproducible

on stock upstream (see "Performance breakdown" under Throughput).

• Hardware: 1× Intel Arc Pro B70 — 256 Xe cores, 30.3 GiB, 2800 MHz, 230 W cap (Level-Zero).
• All benchmark scripts, raw TSVs, and per-request JSON are in this repository's release artifacts.

---

Limitations & caveats

• Hardware focus: the recipe and numbers are tuned for the Intel Arc B70 (SYCL). It runs on any

llama.cpp backend (CUDA/Metal/CPU), but the throughput figures are B70-specific.

• Quant trade-off: Q5_K_M is near-lossless but not lossless; for reference-grade fidelity use Q8_0.
• Thinking model: budget tokens for the <think> block; set a sufficient max_tokens.
• Spec route is conditional: the n-gram route is a structured-deep optimization, not a global win.
• Inherits base-model behavior and biases from Qwen/Qwen-AgentWorld-35B-A3B.

---

License & attribution

Released under Apache-2.0, inherited from the base model

Qwen/Qwen-AgentWorld-35B-A3B. Please cite the

original Qwen AgentWorld model. GGUF conversion, agent-augmented imatrix, and B70 tuning by Frosty40.