---

license: apache-2.0

base_model: barozp/Qwen3.8-27B-Opus-Distill-v2

base_model_relation: quantized

library_name: llama.cpp

pipeline_tag: image-text-to-text

tags:

• gguf
• llama.cpp
• qwen
• reasoning
• opus-distill
• vision
• mtp
• imatrix
• quantized
• v2

---

Qwen3.8-27B-Opus-Distill-v2-GGUF

GGUF quantizations of barozp/Qwen3.8-27B-Opus-Distill-v2 --

a bugfix release of Qwen3.8-27B-Opus-Distill ("v1").

Why this release exists

A community member (zxbc2023) reported a

clean, deterministic bug on v1: certain stacked output-format constraints

(e.g. "no prose" + "no markdown") sent the model into a non-converging

self-verification loop, burning the entire token budget with **zero visible

output**. Root cause traced to part of v1's training data being sourced

from a dataset with reconstructed (not verbatim) reasoning traces. This

release retrains on a rebuilt dataset -- every row traced to a verified

genuine Opus source, every row scanned (30-way parallel LLM-judge pass over

the full set) and dropped if it showed the same pathological pattern.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked

constraints):

| Test | v1 | v2 |

|---|---|---|

| "no prose" + "no markdown" | 3000/3000 tokens, 0 visible output | 87/4096 tokens, clean |

| "no prose" + "no comments" | 3000/3000 tokens, 0 visible output | 73/4096 tokens, clean |

| Generalization (different task) | -- | 43/4096 tokens, clean |

All three converge quickly, </think> closes normally -- no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice),

chat template OFF, QUICK mode (--limit 500). Same protocol as v1, so the

delta column is the meaningful signal.

| Task | Metric | Base | v2 | Delta |

|---|---|---:|---:|---:|

| wikitext | word perplexity (lower better) | 8.4335 | 8.3788 | -0.055 |

| mmlu | acc | 0.8494 | 0.8476 | -0.002 |

| hellaswag | acc_norm | 0.7420 | 0.7500 | +0.008 |

| arc_challenge | acc_norm | 0.5880 | 0.6220 | +0.034 |

| gpqa_diamond | acc_norm | 0.2323 | 0.4697 | +0.237 |

General knowledge (MMLU) and language modeling (wikitext) are essentially

unaffected -- in the same noise band as v1. Reasoning (ARC, GPQA) shows a

large, real gain over base, not comparable to Qwen's own published GPQA

number (thinking-mode-on, different harness) -- this is a same-protocol

base-vs-distill delta.

Available quantizations

| File | Size | Use case |

|---|---:|---|

| Qwen3.8-27B-Opus-Distill-v2-BF16.gguf | 54.7 GB | reference / re-quantization source |

| Qwen3.8-27B-Opus-Distill-v2-Q8_0.gguf | 29.0 GB | near-lossless |

| Qwen3.8-27B-Opus-Distill-v2-Q6_K.gguf | 22.4 GB | high quality |

| Qwen3.8-27B-Opus-Distill-v2-Q5_K_M.gguf | 19.5 GB | quality / balanced |

| Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf | 16.8 GB | recommended all-rounder |

| Qwen3.8-27B-Opus-Distill-v2-Q3_K_M.gguf | 13.5 GB | tight VRAM |

| Qwen3.8-27B-Opus-Distill-v2-IQ3_XXS.gguf | 11.4 GB | low-bit, imatrix |

| Qwen3.8-27B-Opus-Distill-v2-IQ2_XXS.gguf | 8.7 GB | very low-bit, imatrix |

| Qwen3.8-27B-Opus-Distill-v2-IQ1_M.gguf | 7.9 GB | extreme low-bit, imatrix |

K-quants (Q8_0-Q3_K_M) need no imatrix. IQ-quants (IQ3_XXS and below)

require one to run at all in current llama.cpp.

Which one to pick:

• Best quality with headroom -> Q6_K or Q8_0
• Best quality/size balance -> Q4_K_M (default recommendation)
• 24 GB card -> Q4_K_M; 16 GB card -> Q3_K_M (partial offload)
• Below that -> IQ quants, accept the quality hit

Imatrix

imatrix.dat in this repo is **reused from

barozp/Qwen3.8-27B-Opus-Distill-GGUF**

(v1) -- same base architecture, so no need to recompute. As with v1, the

native MTP head (blk.64, the nextn.* decoder layer) is never exercised

by a normal forward pass, so it has no imatrix coverage; llama-quantize

pins that block to q4_K instead of failing.

Vision (mmproj)

The vision tower is in Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf (~0.9 GB). Load it alongside any quant for image/video input:

llama-server -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf

Text-only usage does not need mmproj and runs fine without it.

Quick start

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-v2-mmproj-f16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-v2-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on

Training details (source safetensors model)

• Base: Qwen/Qwen3.8-27B -- dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
• Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
• Final validation loss: 0.4334
• Vision + MTP: carried over byte-for-byte from the base checkpoint -- never trained

Source chain

The full Qwen3.8-27B Opus Distill family:

• Qwen/Qwen3.8-27B — base model
• barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF — v2, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 — v2, FP8 build
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit — v2, MLX 4-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit — v2, MLX 8-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2 — v2, BF16 weights
• barozp/Qwen3.8-27B-Opus-Distill-GGUF — v1, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill — v1, BF16 weights

This release: v2, GGUF quants (the card you are reading).

Thanks to zxbc2023 for the bug report that made this release possible.

Related

• barozp/Qwen3.8-27B-Opus-Distill-v2 -- BF16 source model, full card.
• barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 -- block-FP8 safetensors build for sglang / vLLM / transformers; includes paired BF16-vs-FP8 benchmark data.
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit and MLX-8bit -- Apple Silicon MLX builds via mlx-vlm.