pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder-GGUF overview

---

base_model:

- Jackrong/Qwopus3.6-35B-A3B-Coder

- ornith-ai/Ornith-1.5-35B-A3B

- Kwaipilot/KAT-Coder-V2.5-Dev

- Qwen/Qwen-AgentWorld-35B-A3B

base_model_relation: merge

library_name: transformers

tags:

- merge

- ties

- dare

- moe

- qwen

- qwen3.5

- qwen3.6

- causal-lm

- deltanet

- agentic

- reasoning

- code

license: apache-2.0

language:

- en

- zh

pipeline_tag: text-generation

model_type: qwen3_5_moe

---

<div align="center">


</div>

> 🚀 New Release: Looking for my latest work? Check out the new Qwen3.8-KAT-AgentWorld-35B-Coder!

> Instant access is available 100% free for members of my Patreon community.

<div align="center">

!PentaCoder-9B







</div>

A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation.

> [!IMPORTANT]

> Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks.

---

Contents

• Architectural Specifications
• Composition
• Merge Methodology
• Layer-Stratified Policies
• Chat Template
• Available quants
• Generation Parameters
• Lineage
• References

---

Architectural Specifications

| Spec | Value |

| :--- | :---: |

| Total parameters | 35B |

| Active parameters / token | 3B |

| Decoder layers | 40 |

| Routed experts | 256 |

| Shared experts | 1 |

| Attention | Gated DeltaNet hybrid linear attention |

| Merge algorithm | DARE-TIES + sine depth scaling |

Composition

Jackrong/Qwopus3.6-35B-A3B-Coder serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights.

| Model | Role | Task Weight (α) |

| :--- | :--- | :---: |

| Jackrong/Qwopus3.6-35B-A3B-Coder | Base anchor (W₀) | 1.00 |

| ornith-ai/Ornith-1.5-35B-A3B | Donor (D₁) | 0.30 |

| Kwaipilot/KAT-Coder-V2.5-Dev | Donor (D₂) | 0.25 |

| Qwen/Qwen-AgentWorld-35B-A3B | Donor (D₃) | 0.20 |

---

Merge Methodology

For each floating-point parameter, a task delta is computed per donor \\(k\\):

$$

\Delta_k = D_k - W_0

$$

DARE pruning. A Bernoulli mask at retention density \\(p\\) zeroes out low-magnitude updates; surviving values are rescaled by \\(p^{-1}\\):

$$

\tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p)

$$

TIES sign election. A consensus sign \\(\Gamma\\) is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging:

$$

\Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right)

$$

$$

\Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right)}{\sum_{k=1}^K \alpha_k \cdot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon}

$$

Depth-scaled reconstruction. The merged weight is reconstructed as:

$$

W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}}

$$

where the layer scaling factor \\(\lambda(l)\\) across decoder layer index \\(l \in [0, 39]\\) is defined as:

$$

\lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right)

$$

This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers \\((l \in [12, 28])\\).

---

Layer-Stratified Policies

| Parameter Group | Match Substring | Policy | Density (p) | Base Scale (β) |

| :--- | :--- | :---: | :---: | :---: |

| Embeddings / LM head | embed_tokens, lm_head | Linear | — | 1.00 |

| Norms / biases | norm, bias, 1D tensors | Linear | — | 1.00 |

| DeltaNet recurrent state | a_log, dt_bias, conv1d | Linear | — | 1.00 |

| MoE router gate | mlp.gate.weight, block_sparse_moe.gate | Linear | — | 1.00 |

| MoE shared expert | shared_expert | DARE&#8209;TIES | 0.70 | 0.60 |

| Attention projections | attn, rotary, in_proj, out_proj, x_proj | DARE&#8209;TIES | 0.75 | 0.60 |

| Routed experts (×256) | experts, mlp | DARE&#8209;TIES | 0.65 | 0.55 |

• Router protection: Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing.
• DeltaNet stability: Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space.
• MTP removed: Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference.

---

Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

---

Available quants

| Quantization | File | Size |

| :--- | :--- | :---: |

| Q4_K_M | Qwen-35B-A3B-SignOfFour-Coder-Q4_K_M.gguf | 21.5 GB |

| APEX-High-Quality | Qwen-35B-A3B-SignOfFour-Coder-APEX-High-Quality.gguf | 25.1 GB |

| Q5_K_M | Qwen-35B-A3B-SignOfFour-Coder-Q5_K_M.gguf | 25.0 GB |

| Q6_K | Qwen-35B-A3B-SignOfFour-Coder-Q6_K.gguf | 28.7 GB |

| Q8_0 | Qwen-35B-A3B-SignOfFour-Coder-Q8_0.gguf | 36.9 GB |

| BF16 (Split) | Qwen-35B-A3B-SignOfFour-Coder-BF16-*.gguf | 69.4 GB |

> [!TIP]

> Need fast GPU inference? Running a 35B MoE with CPU offloading or on under-provisioned consumer RAM can result in low token throughput. If you lack a local 24 GB+ GPU and want faster inference, you can spin up an hourly instance on RunPod in one click to run llama-server with 100% layer offload. New accounts created through the link below receive bonus compute credits ($5–$500 on your first $10 top-up).

>

> 

>

> Affiliate Disclosure: If you register an account and load funds using this link, I receive a small commission and platform credits from RunPod at no extra cost to you. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!

---

Recommended Generation Parameters

For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.

| Parameter | Coding / Terminal Agent | Creative Reasoning |

| :--- | :---: | :---: |

| Temperature | 0.6 | 1.0 |

| Top-P | 0.95 | 0.95 |

| Min-P | 0.0 | 0.01 |

| Repetition Penalty | off | 1.05 |

---

Lineage

Qwen/Qwen3.6-35B-A3B
└── pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder
    ├── base:  Jackrong/Qwopus3.6-35B-A3B-Coder
    ├── donor: ornith-ai/Ornith-1.5-35B-A3B
    ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev
    └── donor: Qwen/Qwen-AgentWorld-35B-A3B

---

Citation & References

• Jackrong/Qwopus3.6-35B-A3B-Coder
• ornith-ai/Ornith-1.5-35B-A3B
• Kwaipilot/KAT-Coder-V2.5-Dev
• Qwen/Qwen-AgentWorld-35B-A3B
• Improved Chat Template for Qwen 3.x

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2023}
}