michaelw9999/Qwen3.6-35B-A3B-MXFP6-MTP-GGUF overview

---

base_model:

• Qwen/Qwen3.6-35B-A3B

tags:

• mxfp6
• MXFP6
• MTP
• llama.cpp
• gguf
• qwen3.6

---

Changed GGML TYPE to 50, latest mxfp6 build is required

This is an proof of concept/work in progress Qwen3.6-35B-A3B quantized into MXFP6.<BR>

It was quantized with my experimental <A HREF="https://github.com/michaelw9999/advanced-gguf-quantizer">advanced-gguf-quantizer</A> tool.<BR>

<B>This GGUF will ONLY work with llama.cpp.</B><BR>

The CPU only PR is posted on llama.cpp here:<BR>

https://github.com/ggml-org/llama.cpp/pull/22671

That PR runs slowly because it is for an initial CPU only implementation without GPU support.<BR>

<B>You may install the very fast POC CUDA version from my fork:</B><BR>

https://github.com/michaelw9999/llama.cpp/tree/mxfp6-cuda<BR>

To merge into your existing llama.cpp installation:<BR>

git remote add mxfp6 https://github.com/michaelw9999/llama.cpp
git fetch mxfp6
git merge mxfp6/mxfp6-cuda
cmake -B build -DGGML_CUDA=ON
cmake --build build -j

Or to install fresh:

git clone -b mxfp6-cuda https://github.com/michaelw9999/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build -j

NOTICE:

This is my own work and is experimental and unofficial.

The CUDA version is not part of any llama.cpp PR (yet). This is not associated with NVIDIA in anyway.

Very likely, any future MXFP6 design will not be compatible with this implementation.

For Qwen3.6-35B, MXFP6 is almost as fast as NVFP4 on prefill, and is now even faster with MTP.

Using FP8 for activations, it is faster than NVFP4 on tokengen.

Feedback is both requested and encouraged so I can make further improvements into future llama.cpp PRs.

MXFP6: Final estimate: PPL = 6.7890 +/- 0.04420

(without MTP)
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35moe 35B.A3B MXFP6 - E2M3 |  26.46 GiB |    34.66 B | CUDA       |  99 |           pp512 |      8094.43 ± 49.53 |
| qwen35moe 35B.A3B MXFP6 - E2M3 |  26.46 GiB |    34.66 B | CUDA       |  99 |           tg128 |        188.10 ± 3.20 |

  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32606 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35moe 35B.A3B NVFP4        |  21.48 GiB |    34.66 B | CUDA       |  99 |           pp512 |      8220.18 ± 57.89 |
| qwen35moe 35B.A3B NVFP4        |  21.48 GiB |    34.66 B | CUDA       |  99 |           tg128 |        159.53 ± 0.82 |