---

library_name: transformers

license: apache-2.0

pipeline_tag: text-generation

tags:

• quest
• text-generation
• qwen
• qwen3
• qwen3.5
• moe
• distillation
• chain-of-thought
• imatrix
• GGUF
• mxfp4
• quantized

base_model:

• osunlp/QUEST-35B-RL

---

💎 QUEST-35B-RL-A3B - Custom Mixed Precision GGUFs with Imatrix

> QUEST, a family of open models ranging from 2B to 35B that serve as general-purpose deep research agents designed to handle a wide range of search tasks, with strong capabilities in fact seeking, citation grounding, and report synthesis.\

> QUEST 35B-class MoE full model after mid-training → SFT → RL (Qwen3.5-35B-A3B base, Qwen3_5MoeForConditionalGeneration). Flagship model of the QUEST family.




This repository contains custom, highly optimized, multi-tier mixed precision GGUF weights for osunlp/QUEST-35B-RL.

This model specialises in deep research and analysis. It's excellent to use for the planning & verification stages of agentic tasks.\

For performing other tasks such as programming & implementation, switch to a different model, like Ornith-1.0-35B-A3B.

To learn more about QUEST, read their blog post.\

To learn more about how to use QUEST 35B RL, view the base model.

These quants were generated using manual layer targeting to maximize quality while shrinking the massive VRAM footprint of the Mixture of Experts layers.

📄 GGUF Files

In order of quality:

| Filename | Size | Quants |

| :--- | :--- | :--- |

| QUEST-35B-RL-A3B-MXFP4_MOE_Q8_0_F16-Imatrix.gguf | 20.7 GB | MXFP4_MOE + Q8_0 + F16 |

| QUEST-35B-RL-A3B-MXFP4_MOE_Q8_0-Imatrix.gguf | 19.8 GB | MXFP4_MOE + Q8_0 |

📊 Importance Matrix (Imatrix)

<details>

<summary>Expand to view datasets & details</summary>

The following datasets were used for the imatrix:

• I am currently remaking this imatrix with 2048 ctx, 3000 chunks, with smaller sets of text

• Old/Current: Custom Target Matrix (500 chunks), up to a max of 20MB of each:

- eaddario/imatrix-calibration - tools_huge, code_huge, math_medium

- Glint-Research/Fable-5-traces

- lordx64/fable-sft-combined-v2

- osunlp/QUEST-RL-Data

- osunlp/QUEST-SFT-Data-Open-ended

- osunlp/QUEST-SFT-Data-Objective

</details>

---

🔍 Precision Matrix & Flavor Variations

Standard global quantization presets (like stock MXFP4_MOE) compress the backbone layers uniformly, which degrades the delicate reasoning capabilities of advanced agent models.\

This repository provides two distinct manual configuration layouts to balance precision and memory constraints:

1. The Tri-Quant Hybrid Flavor (MXFP4 + Q8_0 + F16)

QUEST-35B-RL-A3B-MXFP4_MOE_Q8_0_F16-Imatrix.gguf - Designed for maximum quality preservation, this layout implements a strict 3-Tier Precision Matrix:

• Tier 1 (Core & Mamba Gating - F16 Precision):

- token_embd.weight, output.weight - Protects the critical input/output vocabulary mappings. Dramatically prevents text degradation.

- ssm_alpha, ssm_beta - Protects the integrity of the Mamba state-space calculations across long-range context tokens.

• Tier 2 (Backbone & Shared - Q8_0 Precision): ssm_out, *._shexp - Keeps the attention mechanics, and all trailing shared experts at high quality, to protect the logical research loops.
• Tier 3 (Routed Experts - MXFP4 Precision): ffn_down_exps, ffn_gate_exps, ffn_up_exps - Shrink the massive background expert parameters directly to MXFP4.

2. The Dual-Quant Hybrid Flavor (MXFP4 + Q8_0)

QUEST-35B-RL-A3B-MXFP4_MOE_Q8_0-Imatrix.gguf - Designed for a slightly leaner memory profile, this layout utilizes 2-Tier Precision:

• Tier 1 (Backbone - Q8_0 Precision): All attention blocks, Mamba structures, vocabulary embeddings, and internal routers use the universal Q8_0 format.
• Tier 2 (Experts - MXFP4 Precision): The heavy sparse expert blocks are target-quantized directly to MXFP4.

---

📝 Exact Conversion Details

These files were converted via llama-quantize utilizing the following manual recipe parameters:

Convert SafeTensors to GGUF:

# Requires python3.12, with `pip install --upgrade transformers`
python convert_hf_to_gguf.py "QUEST-35B-RL/" --outtype f16 --outfile "QUEST-35B-RL-F16.gguf"

Generate Tri-Quant MXFP4_MOE + Q8_0 + F16:

llama-quantize \
  --tensor-type ".*_shexp\.weight=Q8_0" \
  --tensor-type "token_embd\.weight=F16" \
  --tensor-type "^output\.weight=F16" \
  --tensor-type "blk\..*\.(ssm_alpha|ssm_beta)\.weight=F16" \
  --tensor-type "blk\..*\.(ffn_down_exps|ffn_gate_exps|ffn_up_exps)\.weight=MXFP4" \
  --imatrix "imatrix.gguf" \
  "QUEST-35B-RL-F16.gguf" \
  "QUEST-35B-RL-A3B-MXFP4_MOE_Q8_0_F16-Imatrix.gguf" \
  Q8_0

Generate Dual-Quant MXFP4_MOE + Q8_0:

llama-quantize \
  --tensor-type ".*_shexp\.weight=Q8_0" \
  --tensor-type "blk\..*\.(ffn_down_exps|ffn_gate_exps|ffn_up_exps)\.weight=MXFP4" \
  --imatrix "imatrix.gguf" \
  "QUEST-35B-RL-F16.gguf" \
  "QUEST-35B-RL-A3B-MXFP4_MOE_Q8_0-Imatrix.gguf" \
  Q8_0

---

📝 Local Deployment & llama-server Configuration (config.ini)

Use the following server parameter recommendations (Similar to other Qwen 3.5+ configurations).

# --- Samplers (Dynamic & Expressive) ---
temperature = 0.30
top-k = 15
top-p = 0.90
min-p = 0.05
# --- Penalties (Prevent Syntax & Reasoner Corruption) ---
repeat-penalty = 1.00
presence-penalty = 0.00
# --- DRY Sampler (Protects Indentation & Structural Boilerplate) ---
dry-multiplier = 0.8
dry-base = 1.75
dry-allowed-length = 8
dry-penalty-last-n = 1024
dry-sequence-breaker = ["\n", ":", " ", "\t", "\"", ","]
# --- Enforced Execution Graph ---
samplers = temp;top_k;top_p;min_p;dry

Keep reasoning off for this model.

reasoning = off
reasoning-budget = 0
reasoning-format = none

This works well with 128k context window. It starts to struggle above this, although it may be doable with tighter sampling.

---

ℹ️ Misc Details

I'm doing this as a side hobby, with my AMD 5900X, 64GB DDR4, RTX 3060 12GB & RTX 5060 Ti 16GB.

In addition to the above configuration, I also use:

slots = 1
parallel = 1
no-warmup = true

flash-attn = on
mlock = false
no-mmap = false
no-context-shift = true

batch-size = 2048
ubatch-size = 256

fit = on
fit-target = 768
main-gpu = 0
split-mode = layer
n-gpu-layers = 999
n-cpu-moe = 0
tensor-split = 16,12
override-tensor = (token_embd)=CUDA0,(vision|vpm|nextn)=CPU

cache-type-k = q8_0
cache-type-v = q8_0

jinja = true
chat-template = jinja
chat-template-file = chat_template.jinja

For further quality and better ssm behaviour, this configuration can help:

context-shift = false
cache-type-k = f16
cache-type-v = f16

---

🤝 Support the Journey

As a passionate developer, I'm always programming, automating, or experimenting with new ideas.\

I love building open-source tools, trying out new web tech, and creating things that don't yet exist, including local AI & quantizing models.

I love sharing these creations to give back to the community.\

If my projects have saved you time or helped you out, consider supporting my work below!

👉 Support me on Ko-fi

---

✨ Acknowledgments

• OSU NLP Group for the exceptional QUEST-35B-RL base model.
• noctrex for how I found out about this model, which gave me the inspiration for this.

📜 License

Released under the Apache License 2.0.

🔗 Citation

@misc{xie2026quest,
  title={QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks},
  author={Xie, Jian and Lin, Tianhe and Wang, Zilu and Ning, Yuting and Yao, Yuekun and Xue, Tianci and Zhang, Zhehao and Li, Zhongyang and Zhang, Kai and Wu, Yufan and Chen, Shijie and Gou, Boyu and Han, Mingzhe and Wang, Yifei and Lee, Vint and Wei, Xinpeng and Wang, Xiangjun and Su, Yu and Sun, Huan},
  journal={arXiv preprint arXiv:2605.24218},
  year={2026}
}