Qwen3.6-35B-A3B · uint4_wo_32 → GGUF Q4_1

Base model

| Item | Value |

|----|-----|

| Model | Qwen/Qwen3.6-35B-A3B |

| Architecture | Qwen3_5MoeForConditionalGeneration (MoE + linear_attn + self_attn) |

| Local HF snapshot | ~/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/995ad96eacd98c81ed38be0c5b274b04031597b0 |

Quantization

| Item | Value |

|----|-----|

| Tool | Quark RTN (GPU / ROCm Strix Halo) |

| Scheme | uint4_wo_32 (asymmetric uint4, group_size=32) |

| Quantization date | 2026-07-20 |

| Quantized modules | MoE experts, self_attn, linear_attn |

| Kept in FP16 | mlp.gate, shared_expert, lm_head, visual, mtp |

| Quark checkpoint | uint4-wo32/model.safetensors (~22 GB) |

Reproduce quantization

This directory provides a one-shot entry point for Q4_1: after Quark uint4_wo_32 quantization, the GGUF Q4_1 is exported directly from memory (by default no intermediate safetensors is written to disk).

cd /home/l/work/quantization_work/quark_llama_cpp_test/models/qwen3.6-35b-a3b
./scripts/quantize_q4_1.py

By default the GGUF is written to gguf-q4_1/. You can override the default paths and device via BASE_MODEL, GGUF_OUT, QUARK_SRC, LLAMA_CPP_DIR, DEVICE, MULTI_GPU=1. To quantize only and skip GGUF export, add --skip-gguf-export.

GGUF export

| Item | Value |

|----|-----|

| Format | Q4_1 (quark.torch.export_gguf, export_format="q4_1") |

| Export date | 2026-07-21 |

| Shards | 3 × ~7 GB (split_max_size=8G) |

| Path | gguf-q4_1/ |

| Inference stack | llama.cpp build-hip (ROCm) |

Export paths (hybrid, experimental)

| Path | Tensor count (from log) | Notes |

|------|-------------------|------|

| quark-native --> Q4_1 | 160 | Direct packing, no libggml recompute |

| float32 --> Q4_1 | 150 | linear_attn etc. need V-head reordering: dequant → reshape → re-quantize to Q4_1 |

| float16 --> Q4_1 | 120 | shared_expert and other layers left unquantized |

> Known limitation: linear_attn / shared_expert are re-quantized during export; a future improvement is to make the whole path native passthrough.

Files

gguf-q4_1/
├── qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00001-of-00003.gguf  (7.4 GB)
├── qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00002-of-00003.gguf  (7.4 GB)
└── qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00003-of-00003.gguf  (6.9 GB)

Evaluation: GSM8K

| Item | Value |

|----|-----|

| Date | 2026-07-21 |

| Task | GSM8K test (1319 questions) |

| Protocol | gsm8k_nothink (aligned with hongweimeng / lm-eval) |

| Few-shot | 5-shot |

| Inference | llama-server + /v1/completions |

| Prompt | Answer: <think>\n\n</think>\n\n to suppress the thinking preamble |

| Stop | Question:, </s>, <|im_end|> |

| max_tokens | 256 |

| temperature | 0.0 |

| Scoring | lm-eval flexible-extract / strict-match |

| Raw results | gsm8k_results_nothink.json |

| Reproduction script | scripts/run_gsm8k_nothink.sh |

Results

| Metric | Value |

|------|------|

| flexible-extract | 91.81% (1211 / 1319) |

| strict-match | 91.51% (1207 / 1319) |

| Total time | 5041 s (~84 min) |

| Throughput | 0.262 q/s, 33.1 tok/s (completion) |

| Total output tokens | 166,792 (~126 tok/question) |

Comparison: old unaligned protocol

The old script used a plain Answer: completion with max_tokens=512 and scored on the last number in the generated text. Under that protocol the result was 70.66% (931 / 1319), mainly affected by Qwen3.6's thinking output and the scoring method; it is not a fair direct comparison against the gsm8k_nothink result.

Reproduce

cd /home/l/work/quantization_work/quark_llama_cpp_test/models/qwen3.6-35b-a3b
./scripts/run_gsm8k_nothink.sh

# smoke test example
NUM_QUESTIONS=10 ./scripts/run_gsm8k_nothink.sh

lm-eval compatible config: scripts/gsm8k_nothink.yaml. Logs: logs/gsm8k_eval.log, logs/llama_server_gsm8k.log.

Inference

PROFILE=qwen3.6-35b ./run_llama.sh

# or specify a shard
GGUF=gguf-q4_1/qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00001-of-00003.gguf ./run_llama.sh

2026-07-21 single-machine smoke test: llama-cli loaded successfully, generating ~45 tok/s.

Notes

• The uint4-wo32/ directory here is currently a symlink to an earlier quantization artifact; the GGUF and evaluation results in this directory are authoritative.
• The GSM8K score is a Q4_1 hybrid export test baseline, not an official BF16 reference; the scoring protocol may differ slightly from lm-eval flexible-extract.
• Qwen3.6 emits thinking output by default, which affects token usage and last-number extraction accuracy.

Evaluation: Perplexity (wikitext-2, llama.cpp)

All three were run with llama.cpp llama-perplexity on wikitext-2 test (n_ctx=512, 20 chunks, same data and same parameters), so they are directly comparable.

| Format | PPL | vs float | Size |

|------|-----|---------|------|

| F16 float (baseline) | 6.61 ±0.23 | — | 71 GB |

| Q4_1 (UINT4 affine) | 6.73 ±0.24 | +1.8% | ~21 GB |

| Q4_0 (INT4 symmetric) | 6.79 ±0.24 | +2.7% | ~21 GB |

> Quantization is nearly lossless: Q4_1 is only +1.8% over F16 and Q4_0 only +2.7%, while the size is compressed to about 30% of the float model.

> Before fixing the linear_attn V-head reordering, Q4_0 PPL was 2,619,116; after the fix it is 6.79.

Evaluation: Open LLM Leaderboard v2

How all 6 tasks are measured on the GGUF

> Update (2026-08-06): An earlier version of this card stated that only the 2 generative tasks (IFEval, MATH-hard) could be scored on the GGUF. That limitation has since been lifted by patching llama-server; the four multiple-choice tasks are now measured directly on this GGUF file (see below).

The six leaderboard-v2 tasks use two scoring methods in lm-eval:

• Generative (generate_until): IFEval, MATH-hard. The model generates text freely, which is then matched against rules or the reference answer. Any backend that can generate can score these.
• Multiple-choice (multiple_choice / loglikelihood): BBH, GPQA, MMLU-Pro, MuSR. Each candidate answer is appended to the question, and the model computes the sum of the per-position log-probabilities of that already-fixed token sequence, then picks the highest-scoring candidate. This requires the backend to return the logprob of the input (prompt) tokens.

Stock llama-server returns logprobs only for tokens the model itself generates, not for prompt tokens (both /v1/completions with echo=true and the native /completion with n_probs were tested), so the four multiple-choice tasks originally could not be scored. This was fixed by patching llama-server on branch hongweimeng/gguf-prompt-logprobs: /v1/completions with echo=true + logprobs now returns the legacy OpenAI logprobs format (text_offset / tokens / token_logprobs / top_logprobs) covering the prompt tokens. With that, lm-eval's gguf (GGUFLM) backend scores all four multiple-choice tasks directly on the GGUF. Logprob correctness was verified against llama-perplexity (matching PPL).

So all 6 leaderboard-v2 tasks are now measured on this GGUF file — no vLLM / safetensors substitution is needed.

Results (updated 2026-08-08, all 6 tasks complete)

Backend for this GGUF: the 2 generative tasks (IFEval, MATH-hard) use lm_eval --model local-completions -> llama-server (/v1/completions, -c 32768 -np 4, num_concurrent=4), measured 2026-08-04. The 4 multiple-choice tasks use lm-eval's gguf (GGUFLM) backend against the patched llama-server (branch hongweimeng/gguf-prompt-logprobs, -np 4 -c 131072), measured 2026-08-06. The reference columns were measured with vLLM (--tasks leaderboard, 2026-07-22).

| Task | Metric | BF16 (base)¹ | UInt4-GS128 (vLLM)¹ | This GGUF (llama.cpp) |

|------|--------|:-----------:|:-------------------:|:-------------------------:|

| IFEval | prompt_level_strict | 0.3235 | 0.3383 | 0.3512 |

| IFEval | inst_level_strict | 0.4580 | 0.4664 | 0.4808 |

| MATH-hard | exact_match | 0.5287 | 0.4856 | 0.5211 |

| BBH | acc_norm | 0.6546 | 0.6270 | 0.6449² |

| GPQA | acc_norm | 0.4337 | 0.3658 | 0.4018² |

| MMLU-Pro | acc | 0.5968 | 0.5824 | 0.5837² |

| MuSR | acc_norm | 0.4339 | 0.4206 | 0.4180² |

On both the generative and the multiple-choice tasks this GGUF tracks the BF16 baseline closely, indicating faithful quantization export.

¹ Reference only, not measured on this GGUF file (BF16 and vLLM numbers from --tasks leaderboard, 2026-07-22). UInt4-GS128 is a group_size=128 sibling model, different from this group_size=32 GGUF; shown for context.

² Multiple-choice / loglikelihood task, measured on this GGUF via the patched llama-server (branch hongweimeng/gguf-prompt-logprobs) + lm-eval gguf backend. BBH/GPQA/MuSR measured 2026-08-06; MMLU-Pro completed 2026-08-08 (~114k loglikelihood requests, 12,032 samples, memory-bandwidth bound at ~0.6 it/s).