---

license: apache-2.0

base_model:

- Qwen/Qwen3.6-27B

- nvidia/Qwen3.6-27B-NVFP4

base_model_relation: quantized

library_name: llama.cpp

pipeline_tag: image-text-to-text

tags:

- gguf

- qwen

- qwen3.6

- qwen3_5

- nvfp4

- mtp

- speculative-decoding

- vision

- vlm

- multimodal

- llama.cpp

- quantized

- blackwell

- rtx-5090

---

Qwen3.6-27B-NVIDIA-NVFP4-MTP-GGUF

GGUF conversion of nvidia/Qwen3.6-27B-NVFP4, preserving NVIDIA NVFP4 tensors, with MTP speculative decoding and a BF16 vision projector. Benchmarked on an RTX 5090 with llama-benchy.

Highlights

• NVFP4 preserved: 193 NVFP4 tensors are preserved from NVIDIA's ModelOpt quantized checkpoint.
• Q4 attention: attention (q/k/v/o) and the linear-attention / DeltaNet projections are quantized to Q4_K (down from the original FP8) to keep this build compact. For those layers kept at Q8_0 for better accuracy — about +3 GB — see the Q8attn variant.
• MTP included: the GGUF keeps the extra MTP layer for draft-mtp speculative decoding.
• Vision supported: includes a BF16 mmproj file for image input.
• RTX 5090 tested: measured with llama-benchy using MTP depth d=3.

Provenance

| Component | Source |

|---|---|

| Base model | Qwen/Qwen3.6-27B |

| NVFP4 source checkpoint | nvidia/Qwen3.6-27B-NVFP4 |

| Runtime target | llama.cpp |

Files

| File | Size | Description |

| ------------------------------------------- | -------------------------------: | ------------------------------------------------------- |

| Qwen3.6-27B-NVIDIA-NVFP4-MTP.gguf | 14.66 GiB / 15,747,650,944 bytes | Main GGUF. NVFP4 tensors preserved; MTP layer included. |

| mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf | ~888 MiB / 931,146,304 bytes | BF16 vision projector for image input. |

llama.cpp example

llama-server \
  -m Qwen3.6-27B-NVIDIA-NVFP4-MTP.gguf \
  --mmproj mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -ngl 999

For text-only use, omit --mmproj mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf.

Benchmarks

Measured on RTX 5090, llama-benchy 0.3.8, MTP depth d=3, 3 runs, generation-latency mode.

| Mode | Prompt / generation length | Prefill (pp) tok/s | Generation (tg) tok/s | Peak generation tok/s |

|---|---:|---:|---:|---:|

| Text-only | pp512 / tg512 | 2676.94 ± 51.88 | 112.69 ± 3.13 | 130.33 ± 7.41 |

| Text-only | pp4096 / tg512 | 3401.62 ± 94.87 | 116.38 ± 3.25 | 136.33 ± 4.50 |

| Vision + mmproj | pp512 / tg512 | 3337.94 ± 382.60 | 111.31 ± 7.18 | 126.33 ± 8.38 |

| Vision + mmproj | pp4096 / tg512 | 3650.71 ± 86.68 | 114.15 ± 3.52 | 131.00 ± 2.94 |

For reference, the no-speculative baseline measured in the same project notes was about 70.75 tok/s on pp512/tg512, so MTP d=3 gives roughly +59% generation throughput in that test.

Model details

| Field | Value |

|---|---|

| GGUF architecture | qwen35 |

| License metadata | apache-2.0 |

| Text layers | 64 |

| MTP layers | 1 |

| qwen35.block_count | 65 |

| qwen35.nextn_predict_layers | 1 |

| Vision projector architecture | clip / qwen3vl_merger |

| Vision projector precision | BF16/F32 |

Tensor composition of the main GGUF

| Tensor type | Count |

|---|---:|

| NVFP4 | 193 |

| Q4_K | 279 |

| Q6_K | 34 |

| F32 | 746 |

| Total | 1252 |

The conversion keeps the original NVFP4 tensors instead of requantizing all weights to K-quants.

Attention (q/k/v/o) and the linear-attention / DeltaNet projections are quantized to Q4_K (the

original checkpoint stored them as FP8), which keeps this build compact; the remaining non-NVFP4

tensors are stored as K-quants or F32 as appropriate. If you want those attention / DeltaNet layers

at Q8_0 for better accuracy — about +3 GB — use the

Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn-GGUF

variant instead.

License and attribution

The upstream models are licensed under Apache-2.0:

• Qwen/Qwen3.6-27B
• nvidia/Qwen3.6-27B-NVFP4

This repository redistributes a GGUF format conversion of the NVIDIA NVFP4 checkpoint under the same Apache-2.0 license. Please also follow the terms and usage guidance of the upstream model cards.

Acknowledgements

Thanks to the Qwen team for the base model, NVIDIA for the NVFP4 ModelOpt checkpoint, and the llama.cpp project for GGUF, NVFP4, MTP, and multimodal runtime support.