---

license: apache-2.0

base_model: barozp/Qwen3.8-27B-Opus-Distill

base_model_relation: quantized

library_name: llama.cpp

pipeline_tag: image-text-to-text

tags:

• gguf
• llama.cpp
• qwen
• reasoning
• opus-distill
• vision
• mtp
• imatrix
• quantized

---

Qwen3.8-27B-Opus-Distill-GGUF

GGUF quantizations of

barozp/Qwen3.8-27B-Opus-Distill —

a Qwen3.8-27B fine-tuned with LoRA on Claude Opus reasoning traces (merged),

with the native vision tower and native MTP head carried over untouched.

Highlights

• Reasoning-distilled, not just quantized. The LoRA was trained on 14,250

Opus chain-of-thought traces and merged into the base weights. Quantization

only converts the weights — the reasoning gains travel with them unchanged.

• Full multimodal. Native vision tower ships as a separate mmproj file

(~0.9 GB). Text-only users can ignore it entirely.

• Native MTP for self-speculative decoding. The model was released with its

MTP head trained in — unlike grafted MTP setups, no approximation involved.

Free speedups on compute-bound hardware.

• imatrix-calibrated. All quants below Q3_K_M use an importance matrix

built from the model's own reasoning-distillation data (see Imatrix).

Known issues

Reasoning loop under stacked output-format constraints. Reported by

zxbc2023 (full writeup, discussion #1).

Combining "no prose" with a second output-format constraint (e.g. `"no

markdown" or "no comments"`) can send this model into a non-converging

self-verification reasoning loop -- it burns the entire token budget with

zero visible output. Fully deterministic and reproducible at temp=0.

Root cause: traced to part of the training data being sourced from

reconstructed (not verbatim) Opus reasoning traces, not a capability gap.

Fixed in barozp/Qwen3.8-27B-Opus-Distill-v2

-- retrained on a rebuilt dataset where every row is traced to a verified

genuine source. If you're hitting this, switch to v2.

Workaround if staying on this version: avoid combining "no prose" with

another format constraint, or raise the generation token budget to >=4096

for constrained code-gen tasks.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: **0-shot, loglikelihood (multiple-choice),

chat template OFF, QUICK mode (--limit 500)**. Base and distill ran with the

identical harness, so the Δ column is the meaningful signal.

| Task | Metric | Base | Distill | Δ |

|---|---|---:|---:|---:|

| wikitext | word perplexity ↓ | 8.434 | 8.344 | −0.09 |

| mmlu | acc | 0.849 | 0.849 | −0.001 |

| hellaswag | acc_norm | 0.742 | 0.740 | −0.002 |

| arc_challenge | acc_norm | 0.588 | 0.630 | +0.042 |

| gpqa_diamond | acc_norm | 0.232 | 0.495 | +0.263 |

Reading the table:

• Reasoning improved (ARC +4.2pt, GPQA +26pt), knowledge stayed flat

(MMLU −0.001) and language modeling stayed flat (wikitext −0.09 ppl).

• GPQA caveat: measured with thinking disabled (loglikelihood) — the base

scores near random (25%) because it gets no chance to deliberate. The +26pt Δ

is a valid same-protocol comparison, but do not compare 0.495 to Qwen's

published 89.2 (measured with thinking ON, different harness).

• ARC-Challenge is saturated for modern models; treat it as continuity with

the Qwen3.6 release — GPQA is the stronger reasoning signal here.

Speed (MTP self-speculative decoding)

Not yet benchmarked for this exact model. On the Qwen3.6 sibling (same MTP

mechanism, grafted there), measured with llama.cpp: **+39% tok/s full offload,

+67% partial offload** with spec-decode ON. Native MTP (this model) is trained

in and typically does at least as well. Guidance:

• Compute-bound (full offload, strong GPU) → enable --spec-type draft-mtp.
• Memory-bandwidth-bound (partial offload) → keep spec off.

Available quantizations

| File | Size | Bits/w | Use case |

|---|---:|---:|---|

| Qwen3.8-27B-Opus-Distill-BF16.gguf | 54.7 GB | 16.0 | reference / re-quantization source |

| Qwen3.8-27B-Opus-Distill-Q8_0.gguf | 29.0 GB | 8.5 | near-lossless |

| Qwen3.8-27B-Opus-Distill-Q6_K.gguf | 22.4 GB | 6.6 | high quality |

| Qwen3.8-27B-Opus-Distill-Q5_K_M.gguf | 19.5 GB | 5.7 | quality / balanced |

| Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf | 16.8 GB | 4.9 | recommended all-rounder |

| Qwen3.8-27B-Opus-Distill-Q3_K_M.gguf | 13.5 GB | 4.0 | tight VRAM |

| Qwen3.8-27B-Opus-Distill-IQ3_XXS.gguf | 11.4 GB | 3.3 | low-bit, imatrix |

| Qwen3.8-27B-Opus-Distill-IQ2_XXS.gguf | 8.7 GB | 2.5 | very low-bit, imatrix |

| Qwen3.8-27B-Opus-Distill-IQ1_M.gguf | 7.9 GB | 2.3 | extreme low-bit, imatrix |

K-quants (Q8_0–Q3_K_M) are plain llama-quantize passes, no imatrix needed.

IQ-quants (IQ3_XXS and below) require an importance matrix to run at all

in current llama.cpp and are built from the one in this repo (see below).

Which one to pick:

• Best quality with headroom → Q6_K or Q8_0
• Best quality/size balance → Q4_K_M (default recommendation)
• 24 GB card → Q4_K_M; 16 GB card → Q3_K_M (partial offload)
• Below that → IQ quants, accept the quality hit

Imatrix

imatrix.dat in this repo (512 samples from

barozp/opus-reasoning-distill-train,

context 512) was used to build the IQ quants above. It applies to any GGUF with

this same architecture — including the base

Qwen/Qwen3.8-27B — so it can be reused

for re-quantization without recomputing it:

llama-quantize --imatrix imatrix.dat model-BF16.gguf model-IQ4_XS.gguf IQ4_XS

Note on IQ1_M: the MTP head (blk.64, the nextn.* decoder layer) is

never exercised by a normal forward pass, so the imatrix has no data for it.

llama-quantize pins that block to q4_K instead of failing, which is why

IQ1_M lands at ~2.3 bits/weight (7.9 GB) rather than the ~1.8 a "pure"

IQ1_M would suggest — the MTP head alone accounts for the difference, the rest

of the model is quantized normally.

Vision (mmproj)

The vision tower is in Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf (~0.9 GB) in

this repo. Load it alongside any quant for image/video input:

llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf

Text-only usage does not need mmproj and runs fine without it.

Quick start

# build llama.cpp with CUDA, then:

# text-only chat
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv

# multimodal server
llama-server -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf --mmproj Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf

# with self-speculative decoding (compute-bound hardware)
llama-cli -m Qwen3.8-27B-Opus-Distill-Q4_K_M.gguf -no-cnv --spec-type draft-mtp -fa on

Training details (source safetensors model)

• Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
• Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
• LoRA targets: attention q/k/v/o_proj on the 16 full-attention layers; FFN

gate/up/down_proj on all 64 layers (Gated-DeltaNet projections untouched)

• Data: barozp/opus-reasoning-distill-train (14,250) + -validation (750, held out)
• Run: 1 epoch (891 steps), lr 1e-4 cosine + 3% warmup, effective batch 16,

MAX_SEQ 4096, bf16, ~5h52m on A100 80GB

• Final validation loss: 0.4647
• Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained

Notes

• Thinking mode is on by default (same as the base model). The GGUF embeds

the chat template; how thinking is toggled depends on the llama.cpp version /

frontend (e.g., LM Studio exposes the setting in its UI).

• Conversion: llama.cpp convert_hf_to_gguf.py from the corrected multimodal

config (nested text_config + vision_config).

• No chaining: every quant was produced directly from the BF16 GGUF, so

errors do not accumulate across the ladder.

Source chain

The full Qwen3.8-27B Opus Distill family:

• Qwen/Qwen3.8-27B — base model
• barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF — v2, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 — v2, FP8 build
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit — v2, MLX 4-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit — v2, MLX 8-bit
• barozp/Qwen3.8-27B-Opus-Distill-v2 — v2, BF16 weights
• barozp/Qwen3.8-27B-Opus-Distill-GGUF — v1, GGUF quants
• barozp/Qwen3.8-27B-Opus-Distill — v1, BF16 weights

This release: v1, GGUF quants (the card you are reading).