---

license: apache-2.0

base_model: Qwen/Qwen3.6-35B-A3B

library_name: transformers

pipeline_tag: text-generation

tags:

• gguf
• qwen3.6
• mixture-of-experts
• expert-pruning
• experimental
• llama.cpp

---

Qwen3.6-35B-A3B 128-Expert Pruned GGUF

> Experimental research artifact. This is a 50% expert-pruned model, not an official Qwen release. The pruning calibration is small and coding/agent biased. Test the Q8 artifact on your workload before trusting a lower quant.

This repository contains Q8 through one-bit-class GGUF variants of Qwen/Qwen3.6-35B-A3B. The routed MoE was reduced from 256 to 128 experts per layer while retaining the original top-k 8 active experts per token. Pruning reduces stored parameters and memory, but it does not reduce the eight active expert matmuls per token.

Quant files

| Quant | Size GiB | SHA-256 | QA | Notes |

|---|---:|---|---|---|

| Q8_0 | 18.392 | b85b3ab29aabd631… | 8/8 (pass) | Reference pruned artifact; retained weights preserve their original Q8 bytes. |

| Q6_K | 14.212 | 7fa673151e2f653f… | 6/8 (fail) | Near-Q8 PPL, but missed 2/8 deterministic acceptance checks. |

| Q5_K_M | 12.347 | 495cc9fe01f33bea… | 8/8 (pass) | Best measured local balance: 8/8 checks and only +1.05% PPL. |

| Q4_K_M | 10.592 | b8fec3c109c561b2… | 7/8 (pass) | Smaller viable option: 7/8 checks and +2.38% PPL. |

| Q3_K_M | 8.434 | 0d4dfa3c58100e69… | 6/8 (fail) | Aggressive; verify on your workload. |

| Q2_K | 6.582 | 602bab193dd82587… | 7/8 (pass) | Passed 7/8 smoke checks, but held-out PPL rose 26.50%. |

| IQ2_XS | 5.396 | 9b631cca3ae23cf7… | 6/8 (fail) | Extreme ~2-bit-class build; imatrix required and workload QA strongly recommended. |

| IQ1_M | 4.312 | 4064257359d35174… | 2/8 (fail) | Experimental one-bit-class endpoint (1.75 bpw nominal for quantized tensors). |

| IQ1_S | 3.952 | c5caaa158eb73055… | 1/8 (fail) | Smallest experimental endpoint (1.56 bpw nominal for quantized tensors). |

GGUF has no conventional Q7_K family, so the ladder follows the supported practical families: Q8, Q6, Q5, Q4, Q3, Q2/IQ2, then IQ1. IQ1_M and IQ1_S are both included because they are distinct one-bit-class formats.

Requantization warning: every file below Q8_0 was produced from the pruned Q8_0 artifact with --allow-requantize, not from F16/BF16. The quantizer itself warns that this can reduce quality more than direct quantization from 16/32-bit weights. A remapped 128-expert importance matrix was used for every lower quant.

What was pruned

This checkpoint is not a direct REAP implementation. REAP motivated the experiment, but the Qwen selection used low router-use frequency because that criterion performed better than REAP in the preceding local Granite-MoE control experiment.

See the REAP paper and official Cerebras implementation for the router-weighted activation-norm method that inspired this work.

The selection procedure was:

1. Run 512 coding/agent calibration tokens through all 40 MoE layers.
2. Record 20,480 layer-token observations and each token's top-8 routed expert IDs.
3. In each layer, prune the 128 experts with the lowest routed top-k frequency; break ties deterministically by original expert ID.
4. Copy retained expert rows and corresponding router rows in ascending original-ID order. Update expert_count from 256 to 128.
5. Preserve the raw quantized bytes of retained expert tensors. Shared experts and all other tensors remain byte-identical to the source Q8.
6. Re-open the output and fully hash-check all 733 tensors before publication.

The exact retained IDs, per-expert route counts, source trace hash, and transformation report are included under method/.

Evaluation

The first table compares the unpruned Q8 baseline, the 50%-pruned Q8, and the existing all-expert mixed-Q2 deployment reference on the same eight deterministic coding, JSON, math, tool-call, and context checks.

| Artifact | Size GiB | Passed | Exact outputs vs baseline | Median decode tok/s | Gate |

|---|---:|---:|---:|---:|---|

| full-q8 | 34.369 | 7/8 | 8/8 | 33.84 | PASS |

| pruned-50-q8 | 18.392 | 8/8 | 7/8 | 104.38 | PASS |

| full-mixed-q2 | 16.930 | 5/8 | 4/8 | 118.01 | FAIL |

Exact-output equality is reported separately from task correctness. Runtime numbers are specific to a local RTX 3060 Ti + RTX 2080 SUPER + RTX 3080 setup; the unpruned Q8 baseline used CPU expert offload, while the smaller models fit across all three GPUs.

The complete per-prompt outputs and timing records are in evaluation/. These checks are a smoke/acceptance suite, not a replacement for HumanEval, BFCL, MMLU, long-context, or safety evaluation.

All-quant functional and speed benchmark

| Artifact | Size GiB | Passed | Exact outputs vs baseline | Median decode tok/s | Gate |

|---|---:|---:|---:|---:|---|

| Q8_0 | 18.392 | 8/8 | 8/8 | 104.11 | PASS |

| Q6_K | 14.212 | 6/8 | 6/8 | 108.44 | FAIL |

| Q5_K_M | 12.347 | 8/8 | 8/8 | 118.28 | PASS |

| Q4_K_M | 10.592 | 7/8 | 7/8 | 125.22 | PASS |

| Q3_K_M | 8.434 | 6/8 | 5/8 | 112.63 | FAIL |

| Q2_K | 6.582 | 7/8 | 6/8 | 124.36 | PASS |

| IQ2_XS | 5.396 | 6/8 | 4/8 | 109.60 | FAIL |

| IQ1_M | 4.312 | 2/8 | 1/8 | 100.48 | FAIL |

| IQ1_S | 3.952 | 1/8 | 1/8 | 114.08 | FAIL |

All functional-speed rows except IQ1_M used the local ik_llama CUDA build. That build aborts on IQ1_M with Unhandled type iq1_m (29), so IQ1_M was bench-tested with the local TurboQuant llama.cpp build instead. Its speed is therefore not directly comparable to the other rows; the engine and full command are preserved in evaluation/quant-ladder-qa-results.json.

Held-out perplexity by quant

Perplexity used the included evaluation/qwen36_ppl_heldout.txt, which is separate from both the pruning router trace and the importance-matrix calibration. Values are most useful as relative degradation within this table, not as a cross-model leaderboard.

| Quant | Size GiB | Held-out PPL | Delta vs Q8 | Estimate error |

|---|---:|---:|---:|---:|

| Q8_0 | 18.392 | 9.7356 | +0.00% | 0.81899 |

| Q6_K | 14.212 | 9.7435 | +0.08% | 0.81747 |

| Q5_K_M | 12.347 | 9.8377 | +1.05% | 0.82701 |

| Q4_K_M | 10.592 | 9.9673 | +2.38% | 0.84195 |

| Q3_K_M | 8.434 | 10.9941 | +12.93% | 0.96516 |

| Q2_K | 6.582 | 12.3158 | +26.50% | 1.09987 |

| IQ2_XS | 5.396 | 14.3985 | +47.90% | 1.27363 |

| IQ1_M | 4.312 | 21.4644 | +120.47% | 1.99295 |

| IQ1_S | 3.952 | 29.1534 | +199.45% | 2.88727 |

Practical selection from this sweep

• Q8_0 is the reference for judging pruning independently of further quantization.
• Q5_K_M is the measured recommendation for this local coding/agent workload: 8/8 checks, +1.05% held-out PPL, and 118.28 median decode tok/s.
• Q4_K_M is the smaller viable choice: 7/8 checks, +2.38% PPL, and 125.22 tok/s.
• Q3 and below are research/space-first builds, not general recommendations; their perplexity degradation is material even when a tiny smoke suite passes.

Usage

llama-server \
  -m Qwen3.6-35B-A3B-128E-Pruned-Q4_K_M.gguf \
  --jinja --n-gpu-layers 99 --ctx-size 32768

Use a recent llama.cpp-compatible runtime with qwen35moe support. The Q8 file is the best reference for deciding whether pruning itself is acceptable; move down the quant ladder only after that comparison.

Limitations

• The 512-token selection trace is small and strongly biased toward coding/agent work.
• Experts rarely used by that trace may be important for languages, domains, knowledge, reasoning styles, or safety behavior absent from calibration.
• There was no recovery fine-tuning after pruning.
• Frequency pruning measures router use, not expert activation magnitude or REAP saliency.
• Halving stored experts does not halve decode compute because top-k remains 8.
• Low-bit requants compound pruning risk with Q8-to-lower-bit requantization error.
• Do not use this experiment for high-stakes decisions without independent evaluation.

Reproducibility files

• quant-manifest.json: source, imatrix, quantizer hashes, commands, sizes, and hashes.
• SHA256SUMS: complete model-file checksums.
• method/expert-selection.json: retained/pruned IDs and route counts for every layer.
• method/pruning-report.json: Q8 rewrite and verification report.
• method/imatrix-remap-report.json: 256-to-128 imatrix transformation report.
• evaluation/: deterministic QA outputs and timing evidence.

License and attribution

The GGUF metadata identifies the base license as Apache-2.0. This derivative keeps that license metadata. The source Q8 metadata credits Unsloth as its quantizer. Review the upstream model card and license before redistribution or use.