pragmaticcs/SignOfFour-GGUF overview

---

base_model:

- Jackrong/Qwopus3.6-35B-A3B-Coder

- ornith-ai/Ornith-1.5-35B-A3B

- Kwaipilot/KAT-Coder-V2.5-Dev

- Qwen/Qwen-AgentWorld-35B-A3B

base_model_relation: merge

library_name: transformers

tags:

- merge

- ties

- dare

- moe

- qwen

- qwen3.5

- qwen3.6

- causal-lm

- deltanet

- agentic

- reasoning

- code

license: apache-2.0

language:

- en

- zh

pipeline_tag: text-generation

model_type: qwen3_5_moe

---

<div align="center">







</div>

A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation.

> [!IMPORTANT]

> Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks.

---

Contents

• Architectural Specifications
• Composition
• Merge Methodology
• Layer-Stratified Policies
• Chat Template
• Generation Parameters
• How to Use
• Lineage
• References

---

Architectural Specifications

| Spec | Value |

| :--- | :---: |

| Total parameters | 35B |

| Active parameters / token | 3B |

| Decoder layers | 40 |

| Routed experts | 256 |

| Shared experts | 1 |

| Attention | Gated DeltaNet hybrid linear attention |

| Merge algorithm | DARE-TIES + sine depth scaling |

Composition

Jackrong/Qwopus3.6-35B-A3B-Coder serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights.

| Model | Role | Task Weight (α) |

| :--- | :--- | :---: |

| Jackrong/Qwopus3.6-35B-A3B-Coder | Base anchor (W₀) | 1.00 |

| ornith-ai/Ornith-1.5-35B-A3B | Donor (D₁) | 0.30 |

| Kwaipilot/KAT-Coder-V2.5-Dev | Donor (D₂) | 0.25 |

| Qwen/Qwen-AgentWorld-35B-A3B | Donor (D₃) | 0.20 |

---

Merge Methodology

For each floating-point parameter, a task delta is computed per donor \\(k\\):

$$

\Delta_k = D_k - W_0

$$

DARE pruning. A Bernoulli mask at retention density \\(p\\) zeroes out low-magnitude updates; surviving values are rescaled by \\(p^{-1}\\):

$$

\tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p)

$$

TIES sign election. A consensus sign \\(\Gamma\\) is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging:

$$

\Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right)

$$

$$

\Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right)}{\sum_{k=1}^K \alpha_k \cdot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon}

$$

Depth-scaled reconstruction. The merged weight is reconstructed as:

$$

W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}}

$$

where the layer scaling factor \\(\lambda(l)\\) across decoder layer index \\(l \in [0, 39]\\) is defined as:

$$

\lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right)

$$

This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers \\((l \in [12, 28])\\).

---

Layer-Stratified Policies

| Parameter Group | Match Substring | Policy | Density (p) | Base Scale (β) |

| :--- | :--- | :---: | :---: | :---: |

| Embeddings / LM head | embed_tokens, lm_head | Linear | — | 1.00 |

| Norms / biases | norm, bias, 1D tensors | Linear | — | 1.00 |

| DeltaNet recurrent state | a_log, dt_bias, conv1d | Linear | — | 1.00 |

| MoE router gate | mlp.gate.weight, block_sparse_moe.gate | Linear | — | 1.00 |

| MoE shared expert | shared_expert | DARE&#8209;TIES | 0.70 | 0.60 |

| Attention projections | attn, rotary, in_proj, out_proj, x_proj | DARE&#8209;TIES | 0.75 | 0.60 |

| Routed experts (×256) | experts, mlp | DARE&#8209;TIES | 0.65 | 0.55 |

• Router protection: Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing.
• DeltaNet stability: Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space.
• MTP removed: Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference.

---

Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

---

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

| Parameter | Coding / Terminal Agent | Creative Reasoning |

| :--- | :---: | :---: |

| Temperature | 0.6 | 1.0 |

| Top-P | 0.95 | 0.95 |

| Min-P | 0.0 | 0.01 |

| Repetition Penalty | off | 1.05 |

---

How to Use

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/SignOfFour"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a precise agentic software engineer. Solve problems concisely."},
    {"role": "user", "content": "Write an asynchronous Python queue consumer with retry backoff."}
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(
    inputs,
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    min_p=0.01,
    do_sample=True,
)

print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

---

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/SignOfFour
    ├── base:  Jackrong/Qwopus3.6-35B-A3B-Coder
    ├── donor: ornith-ai/Ornith-1.5-35B-A3B
    ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev
    └── donor: Qwen/Qwen-AgentWorld-35B-A3B

---

Citation & References

• Jackrong/Qwopus3.6-35B-A3B-Coder
• ornith-ai/Ornith-1.5-35B-A3B
• Kwaipilot/KAT-Coder-V2.5-Dev
• Qwen/Qwen-AgentWorld-35B-A3B
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}