---

library_name: transformers

license: apache-2.0

license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE

pipeline_tag: text-generation

base_model:

• Qwen/Qwen3.6-27B

base_model_relation: quantized

tags:

• gguf
• llama.cpp
• quantized
• nvfp4
• mtp
• qwen
• qwen3.6
• dgx-spark
• blackwell

---

s-batman/Qwen3.6-27B-NVFP4-MTP-GGUF

NVFP4 quantization of Qwen3.6-27B with Multi-Token Prediction (MTP) heads, converted to GGUF format using llama.cpp.

This quantization is specifically optimised for NVIDIA Blackwell consumer/edge GPUs (sm_120/sm_121) such as the RTX 5090 and DGX Spark (GB10). NVFP4 uses NVIDIA's native 4-bit block floating point format with E4M3 scaling, providing significantly faster inference than standard Q4_K quants on Blackwell hardware due to hardware-native dequantization.

Model Creator

Qwen Team (Alibaba Cloud)

Original Model

Qwen/Qwen3.6-27B — MTP variant based on unsloth/Qwen3.6-27B-GGUF

Quantization Details

| Property | Value |

|----------|-------|

| Body weights | NVFP4 (GGML type 40) — 311 tensors |

| MTP heads | Q4_K — 194 tensors |

| Norms/biases | F32 — 360 tensors |

| Total size | ~15 GB (4.60 BPW) |

| Source quantization | Q8_K_XL (Unsloth UD-Q8_K_XL with MTP) |

| Conversion tool | llama.cpp build 9277 (commit 40d5358d3) |

| Conversion command | llama-quantize --allow-requantize --tensor-type nvfp4 input.gguf output.gguf Q4_K |

What is NVFP4?

NVFP4 is NVIDIA's native 4-bit floating point format for Blackwell GPUs. Unlike standard integer quantization (Q4_K, Q5_K, etc.), NVFP4 uses block floating point with E4M3 scale factors and is dequantized directly by the GPU's tensor cores. This means:

• Faster inference: Hardware-native dequantization eliminates the integer-to-float conversion overhead
• Lower memory bandwidth: 4.60 BPW vs 10.47 BPW (Q8_K_XL) — 2.3× less data per token
• Good quality: NVFP4 uses per-sub-block scaling (32 elements per sub-block) which preserves more information than uniform 4-bit quantization

MTP (Multi-Token Prediction)

This model includes the MTP prediction head from Qwen3.6, enabling speculative decoding with draft-mtp in llama.cpp. The MTP head is kept in Q4_K to preserve draft quality while the body uses NVFP4 for maximum throughput.

Performance

Benchmarked on NVIDIA DGX Spark (GB10, sm_121, 128 GB LPDDR5X, 273 GB/s bandwidth):

| Quantization | Size | tok/s (diverse prompts) | Draft Acceptance |

|-------------|------|------------------------|------------------|

| Q8_K_XL + MTP | 34 GB | ~13.6 | 0.61-0.82 |

| NVFP4 + MTP | 15 GB | ~27 | 0.41-0.82 |

The 2× speedup comes directly from the reduced memory bandwidth requirement — the GB10's 273 GB/s LPDDR5X is the bottleneck for dense models, and NVFP4 halves the data transfer per token.

Provided Files

| Name | Quant Method | Size | Description |

|------|-------------|------|-------------|

| Qwen3.6-27B-NVFP4-MTP.gguf | NVFP4 (body) + Q4_K (MTP) | ~15 GB | Recommended for Blackwell GPUs |

Usage with llama.cpp

Requirements

• llama.cpp build 8967 or later (NVFP4 support merged in PR #22673)
• CUDA toolkit with Blackwell (sm_120/sm_121) support
• Build with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121 (adjust for your GPU)

Server

llama-server \
  -m Qwen3.6-27B-NVFP4-MTP.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 262144 \
  -ngl 99 \
  -np 1 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --kv-unified \
  --no-mmap \
  --mlock \
  --cont-batching \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 3 \
  --spec-ngram-mod-n-match 24 \
  --spec-ngram-mod-n-min 4 \
  --spec-ngram-mod-n-max 48 \
  --temp 0.6 \
  --top-p 1 \
  --top-k 20 \
  --min-p 0.01 \
  --repeat-penalty 1.1

CLI

llama-cli \
  -m Qwen3.6-27B-NVFP4-MTP.gguf \
  -p "Explain quantum computing in simple terms" \
  -ngl 99 \
  --temp 0.6 \
  --top-p 1 \
  --top-k 20 \
  --min-p 0.01

Download with llama.cpp

llama-cli --hf-repo s-batman/Qwen3.6-27B-NVFP4-MTP-GGUF --hf-file Qwen3.6-27B-NVFP4-MTP.gguf -p "Hello"

Important Notes

• Blackwell only: NVFP4 is a hardware-specific format. It will not run efficiently on non-Blackwell GPUs. For AMD, Intel, or older NVIDIA GPUs, use standard quantizations (Q4_K_M, Q5_K_M, etc.) from unsloth/Qwen3.6-27B-GGUF.
• --no-mmap recommended: On unified memory architectures (DGX Spark), mmap can cause severe slowdowns.
• -np 1 required for MTP: Multi-token prediction speculative decoding currently requires single-parallel mode.

Licensing

This model is licensed under Apache 2.0, same as the original Qwen3.6-27B model.

See LICENSE for details.