---

base_model: pragmaticcs/Triumvirate

quantized_by: pragmaticcs

base_model_relation: quantized

library_name: transformers

tags:

- gguf

- llama.cpp

- merge

- ties

- dare

- moe

- qwen

- qwen3.5

- qwen3.6

- causal-lm

- deltanet

- agentic

- reasoning

- code

license: apache-2.0

language:

- en

- zh

pipeline_tag: text-generation

model_type: qwen3_5_moe

---

<div align="center">

<img src="https://huggingface.co/pragmaticcs/Triumvirate/resolve/main/assets/triumvirate-banner.png" width="100%" alt="Triumvirate Banner" style="border-radius: 8px;" />

</div>

<div align="center">








</div>

Triumvirate (GGUF)

This repository provides quantized GGUF binaries of pragmaticcs/Triumvirate, a 35B MoE (~3.1B active parameters per token) merge built for agentic coding, terminal tool use, and multi-turn repository workflows.

It pairs the strict patch syntax and low tool-error rate of KAT-Coder-V2.5-Dev with algorithmic reasoning from Ornith-1.5 and environment state tracking from Qwen-AgentWorld. Fused via depth-modulated DARE-TIES, it focuses on generating clean search/replace diffs, parsing terminal error feedback, and staying on-task inside autonomous agent loops without conversational drift.

> [!TIP]

> Intended Use: Optimized for agent harnesses. It prioritizes valid tool invocations, concise step-by-step reasoning, and exact code modifications over conversational filler.

---

Contents

• Available Quants
• How to Run
• Recommended Generation Parameters
• Architectural Specifications
• Composition & Triumvirate Roles
• Merge Methodology
• Depth Modulation Dynamics
• Layer-Stratified Policies
• Chat Template
• Lineage
• Citation & References

---

Available Quants

| Quantization | Recommended Use Case | File | Size |

| :--- | :--- | :--- | :---: |

| Q4_K_M | Low VRAM / 24 GB single-card systems | Triumvirate-Q4_K_M.gguf | ~21.5 GB |

| APEX-High-Quality | High fidelity with optimized expert matrix compression | Triumvirate-APEX-High-Quality.gguf | ~25.1 GB |

| Q5_K_M | Recommended Sweet Spot (near-lossless coding diffs) | Triumvirate-Q5_K_M.gguf | ~25.0 GB |

| Q8_0 | Archival / Reference quantization | Triumvirate-Q8_0.gguf | ~36.9 GB |

| BF16 (Split) | Full-precision unquantized baseline | Triumvirate-BF16-*.gguf | ~69.4 GB |

> [!TIP]

> Need fast GPU inference? Running a 35B MoE with CPU offloading or on under-provisioned consumer RAM can result in low token throughput. If you lack a local 24 GB+ GPU and want faster inference, you can spin up an hourly instance on RunPod in one click to run llama-server with 100% layer offload. New accounts created through the link below receive bonus compute credits ($5–$500 on your first $10 top-up).

>

> 

>

> Affiliate Disclosure: If you register an account and load funds using this link, I receive a small commission and platform credits from RunPod at no extra cost to you. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!

---

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

| Parameter | Coding & Terminal Agents | Deep Mathematical Reasoning | Creative Agent Trajectories |

| :--- | :---: | :---: | :---: |

| Temperature | 0.55 | 0.70 | 0.85 |

| Top-P | 0.95 | 0.95 | 0.95 |

| Min-P | 0.01 | 0.01 | 0.02 |

| Repetition Penalty | 1.00 (off) | 1.05 | 1.05 |

---

Architectural Specifications

| Spec | Value |

| :--- | :---: |

| Total Parameters | ~34.1B |

| Active Parameters / Token | ~3.1B |

| Decoder Layers | 40 |

| Routed Experts | 256 (Top-8 routing) |

| Shared Experts | 1 (Active across all tokens) |

| Attention Mechanism | Hybrid: 30 Gated DeltaNet (Linear Attention) + 10 Full Self-Attention layers |

| Hidden Dimension | 2048 |

| MoE Intermediate Size | 512 (Routed) / 1024 (Shared) |

| Vocabulary Size | 248,320 |

| Merge Algorithm | Depth-Modulated MoE DARE-TIES with Calibrated Gate Harmonization |

---

Composition & Triumvirate Roles

| Model | Role | Functional Contribution | Scaling Policy |

| :--- | :--- | :--- | :---: |

| Kwaipilot/KAT-Coder-V2.5-Dev | Base Anchor (W₀) | Code synthesis, strict localized diffs, low tool execution syntax error rate | ≥ 28% structural floor |

| ornith-ai/Ornith-1.5-35B-A3B | Reasoning Donor (Dₒ) | Deep algorithmic planning, proof verification, mathematical search | Dynamic mid-layer peak (λₒ ≤ 0.65) |

| Qwen/Qwen-AgentWorld-35B-A3B | Agentic Donor (Dₐ) | Long-horizon environment simulation, tool-calling state recovery | Transition-layer pulse (λₐ ≤ 0.25) |

---

Merge Methodology

For each parameter tensor, fine-tuning task deltas are computed relative to the base anchor:

$$

\Delta_k = D_k - W_0, \quad k \in \{o, a\}

$$

1. Shape-Safe DARE Sparsification

For prunable weight matrices (numel > 4096), a Bernoulli mask at drop rate \\(p_{\text{drop}}\\) removes low-magnitude interference while rescaling retained coordinates to preserve expectation:

$$

\tilde{\Delta}_k = \frac{1}{1 - p_{\text{drop}}} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(1 - p_{\text{drop}})

$$

2. TIES Sign Election with Multi-Donor Variance Damping

A sign consensus direction \\(\Gamma\\) is computed across surviving updates:

$$

\Gamma = \operatorname{sgn}\left(\sum_k \lambda_k \tilde{\Delta}_k\right)

$$

Conflicting updates are masked to zero. Coordinates where both donors agree receive a dampening factor of \\(\gamma = 1.25\\) to prevent activation explosion:

$$

\Delta_{\text{consensus}} = \frac{\sum_k \left(\lambda_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right)\right)}{\mathbb{I}\left(\sum \text{agrees} > 1\right) \cdot (\gamma - 1) + 1}

$$

$$

W_{\text{final}} = W_0 + \Delta_{\text{consensus}}

$$

3. Calibrated MoE Router Gate Harmonization

Router gates (mlp.gate.weight) are merged without DARE sparsification, scaling interpolated matrices to maintain the base model's Frobenius norm:

$$

\tilde{W}_k = W_k \cdot \frac{\|W_0\|_F}{\|W_k\|_F + \epsilon}

$$

$$

W_{\text{gate}} = \alpha_b W_0 + \alpha_o \tilde{W}_o + \alpha_a \tilde{W}_a, \quad \text{where } \alpha_k = 0.35 \lambda_k(l), \; \alpha_b = \max\left(0, 1 - (\alpha_o + \alpha_a)\right)

$$

$$

W_{\text{gate, final}} = W_{\text{gate}} \cdot \frac{\|W_0\|_F}{\|W_{\text{gate}}\|_F + \epsilon}

$$

---

Depth Modulation Dynamics

Uniform task addition across all layers disrupts low-level tokenization and high-level output selection. Layer scaling factors \\(\lambda_o(l)\\) and \\(\lambda_a(l)\\) are modulated continuously across decoder layers \\(l \in [0, 39]\\) using normalized depth \\(t = \frac{l}{39} \in [0, 1]\\):

$$

\lambda_o(l) = 0.15 + 0.50 \cdot \sin\left(\pi \cdot t^{0.9}\right)

$$

$$

\lambda_a(l) =

\begin{cases}

0.25 \cdot \sin^2\left(\pi \frac{t - 0.35}{0.50}\right) & \text{for } 0.35 \le t \le 0.85 \\

0.0 & \text{otherwise}

\end{cases}

$$

To prevent the base coding anchor from being diluted, if \\(\lambda_o + \lambda_a > 0.72\\), both donor scales are proportionally normalized to guarantee a minimum base anchor weight of \\(\lambda_k \ge 0.28\\).

Depth Band Allocation

| Layer Range | Primary Dynamic | Functional Objective |

| :--- | :--- | :--- |

| Layers 0 – 13 | KAT-Coder Anchor Dominant (λₖ ≥ 0.70) | Protects low-level lexical syntax and token representations. |

| Layers 14 – 22 | Ornith Peak Reasoning (λₒ → 0.65) | Maximizes algorithmic reasoning and multi-step planning. |

| Layers 23 – 34 | AgentWorld Interaction Pulse (λₐ → 0.25) | Infuses world-model simulation and environment-state tracking. |

| Layers 35 – 39 | Base Anchor Re-assertion (λₖ → 0.80) | Stabilizes output head formatting and diff-syntax termination. |

---

Layer-Stratified Policies

| Parameter Group | Match Identifiers | Policy | Drop Rate (p) | Scaling Policy |

| :--- | :--- | :---: | :---: | :---: |

| LM Head | lm_head.weight | Base Anchor | — | 1.00 W₀ (KAT-Coder) |

| Embeddings & Norm | embed_tokens, model.norm | Static 3-Way Blend | — | 0.70 W₀ + 0.20 Wₒ + 0.10 Wₐ |

| DeltaNet Recurrent State | conv1d, in_proj_ba, a_log, dt_bias | DARE-TIES (Zero-Drop) | 0.00 | [λₒ, λₐ] |

| LayerNorms / 1D Biases | layernorm, *norm.weight, 1D tensors | DARE-TIES (Zero-Drop) | 0.00 | [λₒ, λₐ] |

| MoE Router Gates | mlp.gate.weight | Frobenius Invariant | — | [α_b, α_o, α_a] |

| MoE Shared Expert Gate | mlp.shared_expert_gate.weight | DARE-TIES (Zero-Drop) | 0.00 | [0.5 λₒ, 0.5 λₐ] |

| MoE Shared Expert Projections | mlp.shared_expert.{gate,up,down}_proj | DARE-TIES | 0.40 | [0.5 λₒ, 0.5 λₐ] |

| MoE Routed Experts (×256) | mlp.experts.{e}.{gate,up,down}_proj | DARE-TIES (3D Sliced) | 0.70 | [λₒ, λₐ] |

| Full Attention Projections | self_attn.* (Layer index mod 4 = 3) | DARE-TIES | 0.50 | [0.8 λₒ, 0.8 λₐ] |

| Linear Attention Projections | linear_attn.in_proj_qkvz, out_proj | DARE-TIES | 0.60 | [λₒ, λₐ] |

• Recurrent State Integrity: DeltaNet's temporal convolution (conv1d) and update gates (in_proj_ba) are strictly excluded from DARE dropout (zero dropout, \\(p_{\text{drop}} = 0.0\\)) to eliminate numerical divergence in the linear attention state space.
• Vocabulary Padding Hygiene: Unused trailing vocabulary rows are clamped using nan_to_num to prevent uninitialized padding noise from contaminating token embeddings.

---

Chat Template

This model includes the Improved Chat Template for Qwen 3.x by Olivia Rossi, supporting:

• Dedicated Chain-of-Thought (<thought>...</thought>) reasoning blocks.
• Multi-turn tool calling with strict schema verification.
• Built-in error-recovery heuristics for malformed tool parameters.
• Whitespace trimming and prompt token-waste minimization.

---

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/Triumvirate
    ├── Base Anchor:       Kwaipilot/KAT-Coder-V2.5-Dev
    ├── Reasoning Donor:   ornith-ai/Ornith-1.5-35B-A3B
    ├── Simulation Donor:  Qwen/Qwen-AgentWorld-35B-A3B
    └── Quantization:      pragmaticcs/Triumvirate-GGUF

---

Citation & References

• Kwaipilot/KAT-Coder-V2.5-Dev
• ornith-ai/Ornith-1.5-35B-A3B
• Qwen/Qwen-AgentWorld-35B-A3B
• pragmaticcs/Triumvirate (Safetensors Base)
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}