---

license: apache-2.0

base_model: empero-ai/Qwythos-9B-v2

tags:

- gguf

- llama-cpp

- quantized

- mixed-precision

- reasoning

- coding

- deltanet

- hybrid-attention

- ssm

---

Qwythos-9B-v2-PerfectSplit-GGUF

A mixed-precision GGUF quantization of Qwythos-9B-v2 using a "Perfect Split" strategy. The model's 427 tensors are selectively quantized: critical reasoning pathways (attention, DeltaNet/SSM, embeddings, output head) stay at full precision, while the redundant MLP layers are compressed.

🧬 Architecture Context

Qwythos-9B-v2 is a hybrid architecture combining:

• Standard attention layers (Q/K/V/O projections with QK-norm)
• DeltaNet/SSM recurrence layers (alpha, beta, dt, conv1d, ssm_out gates)
• Standard MLP blocks (gate, up, down projections with 3x expansion ratio)

This hybrid design means different tensor classes have different sensitivity to quantization:

• Attention & SSM gates control information routing and long-range memory. Errors here compound across the 1M-token context window.
• MLP layers contain massive mathematical redundancy due to the 3x expansion ratio. They tolerate aggressive quantization with minimal quality impact.

🎯 The "Perfect Split" Strategy

| Tensor Class | Precision | Count | Rationale |

|--------------|-----------|-------|-----------|

| Token Embeddings (token_embd.weight) | F16 | 1 | First point of contact with input; errors propagate everywhere |

| Output Head (output.weight) | F16 | 1 | Maps hidden states → vocabulary logits; critical for token selection |

| Output Norm (output_norm.weight) | F32 | 1 | Final normalization before logits |

| Attention Projections (attn_q, attn_k, attn_v, attn_output, attn_qkv, attn_gate) | F16 | ~96 | Prevents attention score drift at long context |

| Attention Norms (attn_norm, attn_q_norm, attn_k_norm, post_attention_norm) | F32 | ~64 | Activation scaling stability |

| DeltaNet/SSM (ssm_alpha, ssm_beta, ssm_dt, ssm_conv1d, ssm_out, ssm_a) | F16 | ~96 | The recurrence S_t = α·S_{t-1} + β·(k⊗v) runs millions of steps; errors accumulate multiplicatively |

| SSM Norms (ssm_norm) | F32 | ~32 | Recurrence stability |

| MLP Layers (ffn_gate, ffn_up, ffn_down) | Q5_K_M | 96 | Compressed — high redundancy, low sensitivity |

Final Statistics

| Metric | Value |

|--------|-------|

| Original Size (BF16) | 16.69 GB |

| Final Size | 10.78 GB |

| Compression | 35.4% smaller |

| Average BPW | 10.34 |

| F16 Tensors | 331 |

| Q5_K_M Tensors | 96 |

| Total Tensors | 427 |

| Layers | 32 (0 to 31) — all intact |

Per-Tensor Compression Example (from quantization log)

blk.29.attn_q.weight : bf16 → f16 64.00 MiB → 64.00 MiB (preserved)

blk.29.ssm_out.weight : bf16 → f16 32.00 MiB → 32.00 MiB (preserved)

blk.29.ffn_gate.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%)

blk.29.ffn_up.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%)

blk.29.ffn_down.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%)

📊 Expected Quality Profile

Because the quantization is surgical (only compressing MLP layers), the quality profile differs from uniform quants:

| Capability | Expected Quality | Why |

|------------|------------------|-----|

| Chain-of-Thought reasoning | Excellent | Attention + SSM pathways at F16 preserve reasoning flow |

| Long-context recall (128K+) | Excellent | DeltaNet recurrence at F16 prevents drift |

| Code generation | Very Good | MLP compression has minimal impact on coding patterns |

| Factual knowledge recall | Very Good | Slight loss on obscure facts (MLP stores knowledge) |

| Tool-call / JSON output | Very Good | Output head at F16 preserves format fidelity |

| Overall perplexity | ~0.05 above BF16 | vs. ~0.18 above BF16 for uniform Q5_K_M |

Trade-off: This model is ~4 GB larger than a uniform Q5_K_M (~6.5 GB), but delivers noticeably better quality on reasoning-heavy tasks.

🚀 Usage

With llama.cpp

./llama-cli \
  -m Qwythos-9B-v2-PerfectSplit.gguf \
  --prompt "You are an expert Python developer. Solve this step by step inside  tags.\n\nTask: Implement a thread-safe sliding-window rate limiter." \
  --n-predict 2000 \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.05 \
  --repeat-penalty 1.05 \
  --ctx-size 8192 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn
With LM Studio
Import the GGUF file
Recommended settings:
Temperature: 0.6
Top-K: 20
Top-P: 0.95
Min-P: 0.05
Repeat Penalty: 1.05
Context Size: 8192 (or higher if RAM allows)
KV Cache Type: q8_0 (important — keeps total VRAM usage reasonable)
With Ollama
Create a Modelfile:
FROM ./Qwythos-9B-v2-PerfectSplit.gguf
PARAMETER temperature 0.6
PARAMETER top_k 20
PARAMETER top_p 0.95
PARAMETER min_p 0.05
PARAMETER repeat_penalty 1.05
PARAMETER num_ctx 8192

ollama create qwythos-perfect -f Modelfile
ollama run qwythos-perfect