GraySoft
Projects Models Compare Cloud benchmarks FAQ Download guIDE →
Model Intelligence Sheet

FreedomAISVR/MiMo-V2.6-Distill-Qwen-9B-NVFP4-GGUF overview

MiMo V2.6 Distill Qwen 9B — NVFP4 GGUF NVFP4 quantization of Xiaomi's MiMo V2.6 Distill Qwen 9B, packaged for llama.cpp. What this model is MiMo V2.6 Distill Q…

ggufnvfp4llama.cppmultimodalvisionagenticqwen3.5mambahybrid-architecturexiaomi-mimoquantizedimage-text-to-textbase_model:XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9Bbase_model:quantized:XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9Blicense:mitendpoints_compatibleregion:usconversational

Runs locally from ~875.6 MB disk (4 GB VRAM class GPUs with llama.cpp / guIDE).

Downloads
0
Likes
0
Pipeline
image-text-to-text

Repository Files & Downloads

2 GGUF files detected
Direct downloads for local inference
FileTypeQuantizationSizeLink
mimo-v2.6-distill-qwen-9b-nvfp4.ggufGGUFGGUF5.18 GBDownload
mmproj-mimo-v2.6-distill-qwen-9b-f16.ggufGGUFF16875.6 MBDownload

Model Details

Model IDFreedomAISVR/MiMo-V2.6-Distill-Qwen-9B-NVFP4-GGUF
AuthorFreedomAISVR
Pipelineimage-text-to-text
Licensemit
Base modelXiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
Last modified2026-09-25T00:00:23.000Z

Model README

---

license: mit

base_model: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

tags:

- gguf

- nvfp4

- llama.cpp

- multimodal

- vision

- agentic

- qwen3.5

- mamba

- hybrid-architecture

- xiaomi-mimo

- quantized

library_name: gguf

pipeline_tag: image-text-to-text

---

MiMo-V2.6-Distill-Qwen-9B — NVFP4 GGUF

NVFP4 quantization of Xiaomi's MiMo-V2.6-Distill-Qwen-9B, packaged for llama.cpp.

What this model is

MiMo-V2.6-Distill-Qwen-9B is a 9-billion-parameter agentic vision-language model built by Xiaomi MiMo. It is a supervised fine-tune (SFT) of Qwen3.5-9B on data generated by the larger MiMo-V2.6 model family, covering four target domains:

  • Software engineering / agentic coding
  • General-purpose agent work (tool use, terminal, automation)
  • Visual coding (understanding and writing code from screen/UI images)
  • Cybersecurity

Xiaomi released this SFT checkpoint as an open starting point for research in agentic reinforcement learning — the MiMo team's stated goal is to give the community a base checkpoint rather than a closed final product.

Model tree

Qwen/Qwen3.5-9B                       (base architecture, Alibaba Qwen)
└── XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B   (SFT by Xiaomi MiMo, MIT)
    └── FreedomAISVR/MiMo-V2.6-Distill-Qwen-9B-NVFP4-GGUF   (this repo)

Source: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B · License: MIT · Architecture: Qwen3_5ForConditionalGeneration

Architecture

  • Hybrid Mamba-2 + attention: 32 layers — 24 linear-attention (Mamba-2) layers and 8 full-attention layers (full_attention_interval: 4). The Mamba layers are why 256K context is affordable in VRAM.
  • Vision encoder included (333 vision tensors, patch 16, depth 27) — image and video inputs, shipped separately as mmproj.
  • Context: 262,144 tokens (256K) — max_position_embeddings: 262144, inherited from Qwen3.5-9B.
  • Hidden 4096, 16 heads / 4 KV heads, head_dim 256, intermediate 12288, vocab 248,320, rope θ 10M with 0.25 partial rotary, interleaved MRoPE.
  • Thinking is enabled by default (source chat template) — responses begin with [Start thinking].

Reported results (from Xiaomi's model card)

| Domain | Benchmark | Qwen3.5-9B | MiMo-V2.6-Distill-9B |

|---|---|---|---|

| Code | SWE Verified avg@3 | 60.0 | 61.1 |

| Code | SWE Pro avg@3 | 32.0 | 44.6 |

| Code | MiMo Code (mini) avg@3 | 19.5 | 51.6 |

| Cyber | MiMo Cyber (mini) avg@3 | 5.7 | 31.3 |

| General | Terminal Bench 2.1 | 27.0 | 37.1 |

| General | Toolathlon-Verified | 25.9 | 35.2 |

| General | AutomationBench v1.0.6 | 5.0 | 30.3 |

| Visual | MiMo Visual Coding (mini) | 61.7 | 64.0 |

Trained on a 77.4B-token SFT mixture (27.2B loss-bearing) split across code, cyber, general, and visual data. Figures above are Xiaomi's own reported numbers, not reproduced here.

Quantization

| File | Format | Size | BPW | Tensors |

|---|---|---|---|---|

| mimo-v2.6-distill-qwen-9b-nvfp4.gguf | NVFP4 (E2M1 values, E4M3 per-16 scaling + FP32 tensor scale) | 5.18 GiB (5.56 GB) | 4.96 | 249 NVFP4, 177 F32, 1 Q8_0 (lm_head) |

| mmproj-mimo-v2.6-distill-qwen-9b-f16.gguf | F16 | 876 MiB (0.92 GB) | — | Vision encoder, required for image/video input |

  • BPW is measured against the F16 GGUF (16.69 GiB, 427 tensors), not an idealized FP32 baseline.
  • Norms, embeddings, and per-layer scales stay F32 by design — NVFP4 stores weights only.
  • NVFP4 = GGUF type 40 / llama.cpp ftype 39. Hardware acceleration requires NVIDIA Blackwell (RTX 50 series); older GPUs fall back to slower paths.

Inference performance

Measured on NVIDIA GeForce RTX 5060 Ti 16 GB (Blackwell SM120), CUDA 13.2, llama.cpp build b1868-4fea119d, full GPU offload (-ngl 99), KV cache Q8_0 for both K and V.

| Setting | NVFP4 | MXFP4 sibling |

|---|---|---|

| 128K ctx (-c 131072), short prompt — prompt | 18.1 t/s | 54.9 t/s |

| 128K ctx — generation | 66.8 t/s | 70.3 t/s |

| 256K ctx (-c 262144), 258,256-token prompt — prompt | 1662.5 t/s | 1584.8 t/s |

| 256K ctx — generation | 29.8 t/s | 30.8 t/s |

Notes on these numbers:

  • The 128K prompt figure is from a ~20-token prompt, so it reflects a tiny batch and is not representative of long-prompt throughput.
  • Prompt throughput at 256K reflects large-batch processing of the full 258K-token input.
  • Generation speed drops at 256K because the KV cache is roughly twice as large — expected, not a defect.
  • At 256K the model followed instructions correctly and stayed coherent across the full context.

Load test: ✅ loads and generates · Vision test: ✅ llama-mtmd-cli + mmproj correctly described a test image · Chat template: ✅ embedded (tokenizer.chat_template)

Usage

llama.cpp CLI (text):

llama-cli -m mimo-v2.6-distill-qwen-9b-nvfp4.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99

llama.cpp CLI (vision):

llama-mtmd-cli -m mimo-v2.6-distill-qwen-9b-nvfp4.gguf \
  --mmproj mmproj-mimo-v2.6-distill-qwen-9b-f16.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 \
  --image your_image.png

llama.cpp server:

llama-server -m mimo-v2.6-distill-qwen-9b-nvfp4.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080

llama-cpp-python: requires a wheel built against a llama.cpp with NVFP4 support — stock PyPI wheels will not load this file. No quantization-aware Python package is published for it yet.

For grounding tasks with images, llama.cpp suggests --image-min-tokens 1024 (upstream issue #16842).

Pipeline

# 1. F16 conversion (--no-mtp is required, see below)
python convert_hf_to_gguf.py <source_dir> --outfile model-f16.gguf --outtype f16 --no-mtp
# 2. vision encoder
python convert_hf_to_gguf.py <source_dir> --outfile mmproj-model-f16.gguf --outtype f16 --mmproj
# 3. quantize
llama-quantize model-f16.gguf model-nvfp4.gguf NVFP4

Model tree / provenance

Qwen/Qwen3.5-9B → XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B → this repo (quantization only).

Quantization does not change the model's weights, knowledge, or training — it only re-encodes them at lower precision. It does not improve the model.

Known limitations

  • No MTP head. The source config declares mtp_num_hidden_layers: 1, but the released checkpoint contains no MTP weights, so this GGUF is converted with --no-mtp (block_count: 32). Multi-token prediction is unavailable; this is a property of the released checkpoint, not a conversion loss.
  • Thinking is on by default — the upstream template emits [Start thinking] unless the caller disables it.
  • This is an SFT research checkpoint, not a polished production model; Xiaomi positions it as a starting point for agentic RL.
  • Benchmark numbers are Xiaomi's reported results and were not reproduced during quantization.
  • KV cache at 256K is large — on 16 GB cards, Q8 KV leaves limited headroom for long generations.
  • NVFP4 requires Blackwell for hardware-accelerated inference.

Run FreedomAISVR/MiMo-V2.6-Distill-Qwen-9B-NVFP4-GGUF with guIDE

Download guIDE — the AI-native code editor with local LLM inference and 69 built-in tools.

Download guIDE → · Browse 524k+ models · Compare models

Source: Hugging Face · Compare models