---

license: apache-2.0

base_model: empero-ai/Qwythos-9B-v2

base_model_relation: quantized

language:

• en

pipeline_tag: image-text-to-text

library_name: gguf

tags:

• gguf
• llama.cpp
• quantized
• qwythos
• qwen3.5
• ftpo
• reasoning
• uncensored
• long-context
• 1M-context
• function-calling
• multimodal
• vision

---

<p align="center">

<img src="https://huggingface.co/empero-ai/Qwythos-9B-v2/resolve/main/qwythos_v2.png" alt="Qwythos-9B-v2" width="640"/>

</p>

<p align="center"><b>Empero AI</b></p>

Qwythos-9B-v2-GGUF

GGUF quantizations of empero-ai/Qwythos-9B-v2 for llama.cpp, Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes.

Qwythos-9B-v2 is the new and improved Qwythos — all the deep chain-of-thought reasoning of the base Qwythos, with the looping behavior fixed. The looping/degeneration that showed up under greedy or low-temperature decoding is trained out (6.7% → 0%), the native MTP head is restored, and the identity prompt is cleaned up — while knowledge and reasoning are held at (or above) the base Qwythos level.

The fix uses FTPO (Final-Token Preference Optimization): the exact token that starts a repetition loop is identified and the model is gently trained to prefer coherent alternatives at that one position, leaving the rest of the distribution — and therefore its knowledge and reasoning — untouched.

For full training details, evaluation numbers, and sample generations, see the base model card.

---

What's new vs. the base Qwythos

• 🔁 Looping behavior eliminated — repetition under greedy / low-temp decoding dropped 6.7% → 0%. Greedy decoding is now safe — you no longer need repeat-penalty as a band-aid.
• 🧩 MTP head restored — the native multi-token-prediction module is back in the -MTP- files, so speculative-decoding (--spec-type draft-mtp) works.
• 🧠 Reasoning preserved — MMLU / GSM8K / GPQA / ARC held at or above the base Qwythos level (see the model card).
• 🪪 Cleaner identity — states who it is once, only when asked.
• 🔓 Still intentionally uncensored, still 1M-token context (YaRN), still multimodal-capable (Qwen3.5 vision tower).

---

Files

Normal text weights — trunk only (32 blocks)

| File | Quant | Size | Notes |

|---|---|---|---|

| Qwythos-9B-v2-Q4_K_M.gguf | Q4_K_M | 5.34 GiB / 5.74 GB | recommended default — smallest practical, good quality |

| Qwythos-9B-v2-Q5_K_M.gguf | Q5_K_M | 6.08 GiB / 6.52 GB | balanced quality / size |

| Qwythos-9B-v2-Q6_K.gguf | Q6_K | 6.95 GiB / 7.46 GB | high quality |

| Qwythos-9B-v2-Q8_0.gguf | Q8_0 | 8.87 GiB / 9.53 GB | near-lossless |

| Qwythos-9B-v2-BF16.gguf | BF16 | 16.69 GiB / 17.92 GB | full precision (conversion base) |

If you don't know which to pick, Q4_K_M is the right starting point.

MTP-enabled text weights (33 blocks, nextn_predict_layers = 1)

These embed the restored Qwen3.5-compatible MTP head. Use them with llama.cpp builds that support MTP draft speculation (--spec-type draft-mtp). The MTP matrices are retained at Q8_0 in every quantized variant.

| File | Quant | Size | Notes |

|---|---|---|---|

| Qwythos-9B-v2-MTP-Q4_K_M.gguf | Q4_K_M + MTP | 5.50 GiB / 5.90 GB | recommended MTP default |

| Qwythos-9B-v2-MTP-Q5_K_M.gguf | Q5_K_M + MTP | 6.25 GiB / 6.71 GB | balanced quality / size |

| Qwythos-9B-v2-MTP-Q6_K.gguf | Q6_K + MTP | 7.14 GiB / 7.67 GB | high quality |

| Qwythos-9B-v2-MTP-Q8_0.gguf | Q8_0 + MTP | 9.11 GiB / 9.79 GB | near-lossless |

| Qwythos-9B-v2-MTP-BF16.gguf | BF16 + MTP | 17.14 GiB / 18.41 GB | full precision (conversion base) |

Vision projector — for image input

| File | Size | Notes |

|---|---|---|

| mmproj-Qwythos-9B-v2-BF16.gguf | 0.86 GiB / 0.92 GB | CLIP-style vision encoder + projector at BF16 native precision; required for images, pairs with any text quant above |

The vision tower is inherited unchanged from Qwen3.5-9B — it was frozen through both the base Qwythos SFT and the v2 FTPO fine-tune, so image behavior matches base Qwen3.5-9B. This mmproj is interchangeable with any Qwen3.5-9B mmproj-*.gguf.

---

Hybrid-precision quantization (Gated-DeltaNet / SSM tensors)

Qwythos is a hybrid model — a 3:1 mix of Gated-DeltaNet linear-attention (SSM) blocks and full-attention blocks. The linear-attention state tensors are disproportionately sensitive to low-bit quantization, so the K-quants here keep them at higher precision than the surrounding weights:

| Quant | ssm_alpha | ssm_beta | ssm_out |

|---|---|---|---|

| Q6_K | Q8_0 | Q8_0 | Q8_0 |

| Q5_K_M | Q8_0 | Q8_0 | Q6_K |

| Q4_K_M | Q8_0 | Q8_0 | Q6_K |

The remaining SSM state tensors (ssm_a, ssm_conv1d, ssm_dt, ssm_norm) are kept at F32 by the converter. This preserves the hybrid/SSM blocks for a small (~2–4%) increase in file size over a flat K-quant. Q8_0 and BF16 are uniform and need no overrides.

---

Quick start

llama.cpp

llama-cli \
  -m Qwythos-9B-v2-Q4_K_M.gguf \
  -p "Walk through the biochemistry of how organophosphate nerve agents inhibit acetylcholinesterase." \
  -n 8192 \
  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  -c 16384

Because v2's looping is trained out, --repeat-penalty is now optional and greedy decoding (--temp 0) stays coherent.

Ollama

ollama run hf.co/empero-ai/Qwythos-9B-v2-GGUF:Q4_K_M

LM Studio / jan / KoboldCpp

Drop any .gguf into your runtime's model directory. Qwythos uses the standard Qwen3.5 chat template; modern GGUF runtimes load it automatically from the file.

MTP draft speculation

llama-server \
  -m Qwythos-9B-v2-MTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  -c 16384 --port 8080

MTP support requires a recent llama.cpp build. If your runtime doesn't support MTP yet, use the normal files.

---

Vision (image input)

Download a text quant plus the mmproj-*.gguf, then run llama.cpp's multimodal CLI/server:

llama-mtmd-cli \
  -m Qwythos-9B-v2-Q4_K_M.gguf \
  --mmproj mmproj-Qwythos-9B-v2-BF16.gguf \
  --image ./photo.jpg \
  -p "Describe this image in detail." \
  --temp 0.6 --top-p 0.95 --top-k 20 -c 16384

Honest note: all Qwythos training (base SFT and v2 FTPO) was text-only — the vision tower was never fine-tuned, so image-grounded reasoning inherits base Qwen3.5-9B behavior and has not been independently evaluated for this release.

---

Sampling recommendations

Qwythos is a reasoning model — every response opens with a <think>...</think> block before the answer.

| Parameter | Value |

|---|---|

| temperature | 0.6 |

| top_p | 0.95 |

| top_k | 20 |

| repeat_penalty | 1.05 (optional in v2) |

| max_new_tokens | 16384 |

Unlike the base Qwythos, v2 does not loop under greedy / low-temperature decoding — you can use --temp 0 for deterministic runs without repetition. The 0.6-temperature settings above still match Qwen3.5's official thinking-mode recommendations for best quality.

---

Long context (1M tokens)

The GGUFs ship with YaRN rope-scaling baked in for a 1,048,576-token context window (4× the 262,144 native). Set -c up to 1048576; lower it to reduce KV-cache memory for shorter prompts. A single H100/H200-class GPU comfortably handles 256k–512k; the full 1M typically needs multi-GPU or aggressive KV-cache offload.

---

Conversion & verification

• Converted and quantized with llama.cpp (convert_hf_to_gguf.py, llama-quantize), architecture qwen35, GGUF v3.
• MTP variants: default conversion (33-block, nextn_predict_layers = 1, 15 MTP tensors, MTP matrices pinned Q8_0). Normal variants: --no-mtp (32-block trunk-only). mmproj: --mmproj --outtype bf16.
• Hybrid-precision overrides applied per the table above.
• Structurally verified (arch / block count / nextn key / per-tensor types) and smoke-tested for load + coherent generation.
• shasum -a 256 -c SHA256SUMS covers all 11 artifacts.

---

License & acknowledgements

Apache-2.0, inherited from Qwen3.5-9B. Shared for research and experimentation, as-is.

• Developed and released by Empero AI
• Base model: Qwen3.5-9B (Alibaba Qwen team)
• Looping fixed with FTPO (Final-Token Preference Optimization)
• Quantization: llama.cpp (ggml-org)
• HF model: empero-ai/Qwythos-9B-v2