---

license: apache-2.0

language:

- en

- zh

base_model:

- trohrbaugh/Qwen3.8-27B-heretic-ara

base_model_relation: quantized

library_name: gguf

pipeline_tag: image-text-to-text

tags:

- qwen3.5

- qwen3.8

- heretic

- abliterated

- uncensored

- gguf

- mtp

- vision

- multimodal

- mmproj

- vulkan

- llama-cpp

- rocmfpx

- strix-halo

- rdna3.5

- amd

- conversational

model_type: qwen3.5

quantized_by: cygnal

---

Qwen3.8-27B-heretic-ara — GGUF with MTP + Vision

Quantizations of trohrbaugh/Qwen3.8-27B-heretic-ara with MTP (Multi-Token Prediction) tensors preserved (866 tensors) and vision projector (mmproj) for multimodal image understanding.

Includes ROCmFPX quants for the ROCmFPX fork of llama.cpp — ~42 t/s average decode on AMD Strix Halo (50–75% faster than stock Vulkan).

Files

| File | Size | BPW | Format | Description |

|------|------|-----|--------|-------------|

| Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf | 14 GB | 4.26 | Q4_0_ROCMFP4_FAST | Fastest — ROCmFPX Vulkan, ~42 t/s decode with MTP |

| Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf | 16 GB | 4.83 | Q4_K_M | Stock llama.cpp, ~27–28 t/s decode with MTP |

| Qwen3.8-27B-heretic-ara-Q6_K-MTP.gguf | 21 GB | 6.56 | Q6_K | Higher quality, stock llama.cpp |

| Qwen3.8-27B-heretic-ara-ROCmFP6.gguf | 21 GB | 6.50 | Q6_0_ROCMFPX | Higher quality, ROCmFPX Vulkan |

| mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf | 931 MB | — | BF16 | Vision encoder/projector (shared by all quants) |

Which quant should I use?

• ROCmFP4_FAST (14 GB): Fastest option on RDNA 3.5 (Strix Halo, gfx1151). Requires ROCmFPX fork with -dev Vulkan0. ~42 t/s average decode. Best choice if you have Strix Halo and want maximum speed.
• Q4_K_M (16 GB): Best for stock llama.cpp on any hardware. ~27–28 t/s on Strix Halo. Full vision + MTP support. Best choice for general use.
• Q6_K (21 GB): Higher quality for creative writing and nuanced reasoning. ~30% slower on unified memory due to larger file size. Choose if quality matters more than speed.
• ROCmFP6 (21 GB): Higher quality ROCmFPX variant. ~24 t/s on Strix Halo. Use if you want both ROCmFPX speed and higher quality, and have memory headroom.

Source model

trohrbaugh/Qwen3.8-27B-heretic-ara — Arbitrary-Rank Ablation (ARA) uncensoring of Qwen3.8-27B using heretic v1.2.0+custom. Parameters: start_layer 26, end_layer 56, preserve_good_behavior_weight 0.9432, steer_bad_behavior_weight 0.0009. KL divergence 0.0535, 0/100 refusals.

How to use

Stock llama.cpp — Text-only with MTP (fastest stock decode)

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -ctk f16 -ctv f16 \
  -c 32768 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

Swap in Q6_K-MTP.gguf for higher quality. --spec-draft-n-max 3 is optimal on AMD Strix Halo (Vulkan). Try 4 on other hardware.

Stock llama.cpp — Vision + MTP (multimodal)

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
  -ngl 99 -c 8192 -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --jinja

Then send images via the OpenAI-compatible API:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
      {"type": "text", "text": "Describe this image."}
    ]
  }],
  "max_tokens": 300
}'

Note: mmproj has zero impact on text-only request performance — the vision encoder only activates for image-bearing requests. Safe to load mmproj permanently.

ROCmFPX — Maximum speed (text-only)

Requires ROCmFPX fork built for your GPU.

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 32768 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 6

Critical flags:

• -dev Vulkan0 — forces RADV cooperative matrix path. Without it, ROCm backend runs at ~18 t/s instead of ~42.
• -ctk q8_0 -ctv turbo4 — TurboQuant KV cache. Speed-neutral, dramatically reduces KV memory (enables 262K context in 20 GB vs 61 GB with f16 KV).
• --spec-draft-n-max 6 — optimal for ROCmFPX (vs 3 for stock Vulkan).

ROCmFPX — Vision

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 8192 -np 1 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --jinja

Important: MTP speculative decoding must be disabled for vision requests on ROCmFPX (b213) — the non-consecutive token positions from image embeddings crash MTP boundary tracking. Omit --spec-type draft-mtp when using --mmproj. Vision decode runs at ~13.6 t/s without MTP. Text-only requests will not benefit from MTP in this configuration; use a separate text-only server with MTP for maximum text speed.

Without MTP (slower but works everywhere)

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -c 32768

Benchmarks (AMD Ryzen AI Max+ 395, 122 GB unified, gfx1151)

ROCmFPX (Vulkan0, MTP n-max 6, TurboQuant KV)

| Task | ROCmFP4_FAST Decode | ROCmFP4_FAST Prefill | ROCmFP6 Decode | ROCmFP6 Prefill |

|------|--------------------|--------------------|----------------|-----------------|

| Code gen #1 | 34.5 t/s | 42.0 | 25.7 t/s | 29.5 |

| Code gen #2 | 41.6 t/s | 61.5 | 29.3 t/s | 29.7 |

| Code gen #3 | 49.5 t/s | 61.5 | — | — |

| Math | 37.5 t/s | 97.4 | 22.9 t/s | 50.4 |

| JSON extraction | 44.7 t/s | 137.1 | — | — |

| Technical | 43.8 t/s | 59.6 | 18.1 t/s | 21.4 |

| Average | ~42 t/s | | ~24 t/s* | |

*ROCmFP6 benchmarked under swap pressure (21 GB model at 8K ctx). Would likely hit ~30+ t/s without swap contention.

Stock Vulkan (MTP n-max 3)

| Quant | Prefill (t/s) | Decode (t/s) | BPW |

|-------|--------------|-------------|-----|

| Q4_K_M-MTP | 168–172 | 27–28 | 4.83 |

| Q6_K-MTP | ~65 | ~17–23 | 6.56 |

| Q4_K_M-MTP, no MTP | 200 | 12.2 | 4.83 |

Vision (with mmproj, Q4_K_M, stock Vulkan)

| Config | Prompt tokens | Prefill (t/s) | Decode (t/s) | MTP acceptance |

|--------|--------------|---------------|--------------|----------------|

| Vision + MTP n-max 3 (simple image) | 79 | 23.7 | 13.5 | 62.6% |

| Vision + MTP n-max 3 (complex image) | 830 | 93.4 | 12.7 | 54.6% |

| Vision, no MTP | 79 | 44.9 | 12.1 | — |

Vision decode is ~13 t/s (vs 28 text-only) due to vision encoder overhead. Text-only requests are not affected by having mmproj loaded.

ROCmFPX vision (without MTP): ~13.6 t/s decode, ~85.9 t/s prefill. MTP must be disabled for vision on ROCmFPX b213.

Stock Vulkan benchmarked on llama.cpp build 0ef6e55ed (b10254). ROCmFPX benchmarked on b213/b2f5829.

Speed comparison summary

| Config | Avg Decode (MTP) | Avg Prefill | File Size | Memory fit (122 GB) |

|--------|-----------------|-------------|-----------|---------------------|

| ROCmFPX FP4_FAST | ~42 t/s | ~77 t/s | 14 GB | ✅ Comfortable |

| Stock Q4_K_M | ~27–28 t/s | ~170 t/s | 16 GB | ✅ Comfortable |

| ROCmFPX FP6 | ~24 t/s | ~33 t/s | 21 GB | ⚠️ Swap at 32K |

| Stock Q6_K | ~17–23 t/s | ~65 t/s | 21 GB | ⚠️ Swap at 131K |

Code quality — EvalPlus HumanEval+ (164 problems, pass@1, temp=0)

| Quant | HumanEval | HumanEval+ | BPW |

|-------|-----------|------------|-----|

| Q6_K-MTP | 88.4% | 81.7% | 6.56 |

| Q4_K_M-MTP | 82.3% | 75.6% | 4.83 |

| ROCmFP4_FAST | 75.6% | 71.3% | 4.26 |

| ROCmFP6 | 73.8% | 67.7% | 6.50 |

Stock llama.cpp quants (Q6_K, Q4_K_M) outperform ROCmFPX quants at similar BPW — Q6_K-MTP (6.56 BPW) scores 14% higher on HumanEval+ than ROCmFP6 (6.50 BPW). The ROCmFPX quantization format trades code quality for decode speed. All quants handle math/reasoning, creative writing, and vision correctly on qualitative tests.

Qualitative comparison

All quants tested on identical prompts at temperature 0.0:

• Math/reasoning: All produce correct step-by-step solutions
• Creative writing: All produce strong prose. Q6 variants show marginally richer vocabulary.
• Vision: All correctly identify shapes, colors, and text in test images.

Architecture

Qwen3.8-27B uses the qwen3.5 architecture — a hybrid DeltaNet + attention model (48 linear + 16 full-attention layers) with an integrated vision encoder. Key notes:

• Natively multimodal — 333 vision tensors for image understanding (extracted as mmproj)
• Prompt caching is broken in llama.cpp for hybrid-memory models (every turn reprocesses full context)
• Only 16 layers grow KV cache → cheap context scaling
• MTP provides ~2.3× decode speedup via speculative decoding (text-only, stock Vulkan)
• MTP on ROCmFPX provides ~3.2× decode speedup (13 → 42 t/s)

Quantization details

• ROCmFP4_FAST: Q4_0_ROCMFP4_FAST, 4.26 BPW, ~14 GB, 866 tensors (MTP nextn.* → q8_0 auto)
• Q4_K_M: 4.83 BPW, ~16 GB, 866 tensors (851 model + 15 MTP)
• Q6_K: 6.56 BPW, ~21 GB, 866 tensors (851 model + 15 MTP)
• ROCmFP6: Q6_0_ROCMFPX, 6.50 BPW, ~21 GB, 866 tensors (MTP nextn.* → q8_0 auto)
• mmproj: BF16, 931 MB, 334 tensors (vision encoder — CLIP arch, cannot be quantized)
• Source: BF16 safetensors → BF16 GGUF (51 GB) → all quant variants
• Stock quants: convert_hf_to_gguf.py → llama-quantize
• ROCmFPX quants: convert_hf_to_gguf.py → ROCmFPX llama-quantize (b213/b2f5829)
• mmproj: convert_hf_to_gguf.py --mmproj --outtype bf16

MTP tensors included (all quants)

blk.64.nextn.eh_proj.weight
blk.64.nextn.enorm.weight
blk.64.nextn.hnorm.weight
blk.64.nextn.shared_head_norm.weight
blk.64.nextn.attn_k.weight
blk.64.nextn.attn_k_norm.weight
blk.64.nextn.attn_norm.weight
blk.64.nextn.attn_output.weight
blk.64.nextn.attn_q.weight
blk.64.nextn.attn_q_norm.weight
blk.64.nextn.attn_v.weight
blk.64.nextn.ffn_down.weight
blk.64.nextn.ffn_gate.weight
blk.64.nextn.ffn_up.weight
blk.64.nextn.ssm_out.weight