---

language: [en, zh]

license: apache-2.0

library_name: gguf

base_model: Qwen/Qwen3.5-35B-A3B

tags: [quantized, gguf, mixed-precision, bitclass, bitclass3, qwen3, moe]

pipeline_tag: text-generation

---

Qwen3.5-35B-A3B — BitClass3 Mixed-Precision GGUF

Mixed-precision GGUF quantizations of Qwen3.5-35B-A3B,

a 256-expert (8 routed + 1 shared active) Mixture-of-Experts model with hybrid Gated-DeltaNet + attention

layers (~3B active parameters per token). Each file is produced with llama.cpp's --target-bpw per-tensor

allocator (PR #15550), which assigns a quantization type to every tensor — attention, the 256-way expert

FFN, shared experts, and the DeltaNet/SSM tensors — to hit a target bits-per-weight with minimum

imatrix-weighted error.

These are text-only GGUFs of the language model (the vision tower is not included) with the

NextN / multi-token-prediction head removed — standard text-generation quants for use with llama.cpp.

Available Quantizations

Perplexity (PPL) is measured on wikitext-2 (-c 2048, full file); the BF16 source scores 5.868.

KL-divergence (KLD) against the BF16 source is the primary fidelity metric, reported as the mean and the

robust 99.9th percentile.

| File | BPW | Size | PPL ↓ | PPL/BF16 | KL-mean ↓ | KL-99.9% ↓ | Use Case |

|---|---|---|---|---|---|---|---|

| Qwen3.5-35B-A3B-Q8_0.gguf | 8.44 | 36.56 GB | 5.874 | 1.001 | 0.0054 | 0.157 | Highest-fidelity reference |

| Qwen3.5-35B-A3B-Q6_K.gguf | 6.52 | 28.25 GB | 5.883 | 1.003 | 0.0082 | 0.386 | High quality |

| Qwen3.5-35B-A3B-Q5_K_M.gguf | 5.65 | 24.50 GB | 5.880 | 1.002 | 0.0169 | 0.469 | Balanced quality and size |

| Qwen3.5-35B-A3B-Q4_K_M.gguf | 4.84 | 20.97 GB | 5.911 | 1.007 | 0.0198 | 0.639 | Strong quality-to-size ratio |

| Qwen3.5-35B-A3B-Q3_K_S.gguf | 3.47 | 15.05 GB | 6.544 | 1.115 | 0.1097 | 2.803 | Maximum compression |

Recommended: Q4_K_M sits within ~0.74% of BF16 perplexity at 20.97 GB (KL-mean 0.020). Q5_K_M is

within ~0.2% of BF16 PPL at 24.50 GB. From Q5 upward the levels sit within PPL measurement noise (±0.04)

of each other and of BF16, so they can be chosen by size; KL-divergence keeps falling across that range

(0.017 → 0.008 → 0.005) and is the finer discriminator at the high-bpw end.

Fidelity to the BF16 source

| Model | BPW | Size | wiki PPL ↓ |

|---|---|---|---|

| BF16 (source) | 16.0 | ~69 GB | 5.868 |

| Q8_0 | 8.44 | 36.56 GB | 5.874 |

| Q6_K | 6.52 | 28.25 GB | 5.883 |

| Q5_K_M | 5.65 | 24.50 GB | 5.880 |

| Q4_K_M | 4.84 | 20.97 GB | 5.911 |

| Q3_K_S | 3.47 | 15.05 GB | 6.544 |

Quality is reported against the model's own BF16 source. The file names (Q4_K_M, Q6_K, …) denote the

target tier each mix is built around; the per-tensor allocation is solved to a BPW target and can be

inspected with gguf-dump. A per-tensor type manifest is included in this repo as manifest.md.

Task Accuracy (quantization fidelity)

HellaSwag (2000 tasks) and Winogrande (1267 tasks), scored with llama.cpp log-likelihood on the BF16

source and on each quant under one identical harness. The signal of interest is the delta vs BF16,

which isolates how faithfully each quant preserves the source model's behavior. These are log-likelihood

fidelity measurements, not the model's headline generative capability.

| File | HellaSwag | Δ vs BF16 | Winogrande | Δ vs BF16 |

|---|---|---|---|---|

| BF16 (source) | 81.10 | — | 74.27 | — |

| Q8_0 | 81.25 | +0.15 | 74.98 | +0.71 |

| Q6_K | 81.05 | −0.05 | 74.27 | 0.00 |

| Q5_K_M | 80.80 | −0.30 | 74.27 | 0.00 |

| Q4_K_M | 80.85 | −0.25 | 74.19 | −0.08 |

| Q3_K_S | 79.35 | −1.75 | 74.19 | −0.08 |

From Q4_K_M upward, both tasks sit within measurement noise of BF16 (≈±1.7 HellaSwag / ±2.4 Winogrande,

95% CI at these task counts). Q3_K_S shows a HellaSwag delta of −1.75, near the ±1.7 threshold. This

tracks the PPL/KLD picture: above ~Q4 the quants preserve the source model's task behavior.

Generative reasoning (indicative)

A reproduction of two public benchmarks, run generatively (served with llama-server, thinking on, the

answer parsed from the response) on BF16 and two quants. Every question was scored on a full thinking pass

(up to a 32K-token budget at 40K context); all questions produced a parsed answer.

n = 80 per benchmark (95% CI ≈ ±8–9 points), so these are indicative. The reliable signal is quant-vs-BF16.

Reference figures are Qwen's official Qwen3.5-35B-A3B scores, shown for context (these differ from the

newer Qwen3.6-35B-A3B at 85.2 / 86.0, which is a separate model).

| Model | MMLU-Pro | GPQA-Diamond |

|---|---|---|

| Qwen3.5-35B-A3B (official, reference) | 85.3 | 84.2 |

| BF16 (ours) | 83.8 | 81.2 |

| Q8_0 | 86.2 | 77.5 |

| Q4_K_M | 83.8 | 82.5 |

Across both tasks the quants fall within the ±8–9 point sampling noise of BF16; at n = 80 this test

resolves differences of roughly ±8 points, so it complements (rather than replaces) the much finer

PPL/KLD fidelity measurements above.

How It Works

1. Per-tensor allocation — llama.cpp's --target-bpw solver (PR #15550) assigns a quantization type

to every tensor directly, hitting the target BPW with minimum imatrix-weighted error. This scales to

the MoE's tens of thousands of expert tensors without a hand-written per-layer recipe.

1. imatrix — an importance matrix computed over wikitext guides the per-tensor error weighting.
2. NextN/MTP head removed — the source ships a multi-token-prediction (NextN) head as a trailing

block; it is stripped so these are standard text-generation quants (block_count 41 → 40).

1. Text-only — the vision tower / mmproj is not included; these GGUFs contain the language model only.

This, with the stripped MTP head, accounts for the parameter count being below the upstream repo's

listed total.

Usage

hf download sh111111111111111/Qwen3.5-35B-A3B-BitClass3-GGUF \
    Qwen3.5-35B-A3B-Q4_K_M.gguf --local-dir .

# -c sets context length; raise it toward the model's 262K native limit as RAM allows.
# Sampling below uses Qwen's recommended thinking-mode settings; the presence penalty
# matters for this model.
llama-cli    -m Qwen3.5-35B-A3B-Q4_K_M.gguf -cnv -c 20480 \
    --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5
llama-server -m Qwen3.5-35B-A3B-Q4_K_M.gguf --port 8080 -c 20480 --jinja \
    --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5

> Per Qwen's card, thinking-mode general tasks use `temperature=1.0, top_p=0.95, top_k=20,

> presence_penalty=1.5 (use presence_penalty=0.0` for code). Requires a current llama.cpp build with

> qwen3_5_moe architecture support.

Benchmark Details

NVIDIA GB10 (128 GB unified memory, aarch64). llama.cpp with --target-bpw (PR #15550). The imatrix is

computed from wikitext; wikitext-2 PPL via llama-perplexity (-c 2048, full file); KLD via

llama-perplexity --kl-divergence against the BF16-source logits (mean / median / 99.9th percentile

reported). The BF16 reference is the converted, MTP-stripped GGUF of the language model. Task accuracy via

llama-perplexity --hellaswag (2000 tasks) and --winogrande (1267 tasks). Generative MMLU-Pro /

GPQA-Diamond via llama-server (thinking on) at n = 80 each, as small-sample reproductions.

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen or the llama.cpp project. All benchmark

numbers here are from our own measurements on the hardware and harness described above; Qwen's official

figures, where shown, are labelled as such and are provided for context.

License

Apache 2.0, inherited from Qwen3.5-35B-A3B.