jorge-erdb/qwen3.5-35b-a3b-claude-4.6-opus-reasoning-distilled-4bit-gguf overview

{
  "metadata": {},
  "card_data": {
    "base_model": [
      "qwen/Qwen3.5-35B-A3B",
      "Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled"
    ],
    "tags": [
      "text-generation-inference",
      "transformers",
      "unsloth",
      "qwen3_5_moe",
      "unsloth",
      "qwen",
      "qwen3.5",
      "reasoning",
      "chain-of-thought",
      "4bit",
      "imatrix"
    ],
    "license": "apache-2.0",
    "language": [
      "zh",
      "en",
      "ko"
    ],
    "pipeline_tag": "image-text-to-text",
    "datasets": [
      "nohurry/Opus-4.6-Reasoning-3000x-filtered",
      "Jackrong/Qwen3.5-reasoning-700x"
    ],
    "frontmatter": {
      "base_model": [
        "qwen/Qwen3.5-35B-A3B",
        "Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled"
      ],
      "tags": [
        "text-generation-inference",
        "transformers",
        "unsloth",
        "qwen3_5_moe",
        "unsloth",
        "qwen",
        "qwen3.5",
        "reasoning",
        "chain-of-thought",
        "4bit",
        "imatrix"
      ],
      "license": "apache-2.0",
      "language": [
        "zh",
        "en",
        "ko"
      ],
      "pipeline_tag": "image-text-to-text",
      "datasets": [
        "nohurry/Opus-4.6-Reasoning-3000x-filtered",
        "Jackrong/Qwen3.5-reasoning-700x"
      ]
    },
    "hero_image_url": "https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/GHkMJL6I383eIwK1qj80K.jpeg",
    "summary": "huggingface-cli download jorge-erdb/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-4bit-GGUF --include \"*IQ4_NL.gguf\" --local-dir ./ # Q4_K_M (linear, Metal-friendly) huggingface-cli download jorge-erdb/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-4bit-GGUF --include \"*Q4_K_M.gguf\" --local-dir ./ ``` > [!Important] > ## Apple Metal Backend Warning > > **IQ4_NL is a non-linear quantization format.** It performs sub-optimally on Apple's Metal backend due to the lack of native support for non-linear dequantization kernels. If you are running on an Apple Silicon Mac with GPU offloading via Metal, you will likely experience: > > - Slower inference compared to linear quants of similar size (e.g., Q4_K_M) > - No speed benefit from the ARM weight repacking that IQ4_NL supports on CPU > > **If you're on Apple Metal, use the Q4_K_M quant from this repo instead.** For higher precision options, see Jackrong's repo.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nbase_model:\n- qwen/Qwen3.5-35B-A3B\n- Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled\ntags:\n- text-generation-inference\n- transformers\n- unsloth\n- qwen3_5_moe\n- unsloth\n- qwen\n- qwen3.5\n- reasoning\n- chain-of-thought\n- 4bit\n- imatrix\nlicense: apache-2.0\nlanguage:\n- zh\n- en\n- ko\npipeline_tag: image-text-to-text\ndatasets:\n- nohurry/Opus-4.6-Reasoning-3000x-filtered\n- Jackrong/Qwen3.5-reasoning-700x\n---\n# Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-GGUF — 4-bit\n\n4-bit GGUF quantizations of [Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled](https://huggingface.co/Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled), a Claude 4.6 Opus reasoning-distilled fine-tune of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B).\n\n## Quantization Details\n\n| Detail | Value |\n|---|---|\n| Quant types | IQ4_NL, Q4_K_M |\n| Quantized by | [jorge-erdb](https://huggingface.co/jorge-erdb) |\n| Method | [llama.cpp](https://github.com/ggml-org/llama.cpp) with importance matrix |\n| Importance matrix | [bartowski's imatrix calibration dataset](https://github.com/ggerganov/llama.cpp/discussions/5263) |\n| Source model | [Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled](https://huggingface.co/Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled) (BF16) |\n\n| Quant | Best for | Notes |\n|---|---|---|\n| IQ4_NL | CUDA / CPU | Non-linear, slightly better quality per bit |\n| Q4_K_M | CUDA / CPU / **Metal** | Linear, best Metal compatibility |\n\n## Download\n\n```bash\npip install -U \"huggingface_hub[cli]\"\n\n# IQ4_NL (non-linear, best for CUDA/CPU)\nhuggingface-cli download jorge-erdb/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-4bit-GGUF --include \"*IQ4_NL.gguf\" --local-dir ./\n\n# Q4_K_M (linear, Metal-friendly)\nhuggingface-cli download jorge-erdb/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-4bit-GGUF --include \"*Q4_K_M.gguf\" --local-dir ./\n```\n\n> [!Important]\n> ## Apple Metal Backend Warning\n>\n> **IQ4_NL is a non-linear quantization format.** It performs sub-optimally on Apple's Metal backend due to the lack of native support for non-linear dequantization kernels. If you are running on an Apple Silicon Mac with GPU offloading via Metal, you will likely experience:\n>\n> - Slower inference compared to linear quants of similar size (e.g., Q4_K_M)\n> - No speed benefit from the ARM weight repacking that IQ4_NL supports on CPU\n>\n> **If you're on Apple Metal, use the Q4_K_M quant from this repo instead.** For higher precision options, see [Jackrong's repo](https://huggingface.co/Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled).\n\n## Credits\n\n- **Quantization**: [jorge-erdb](https://huggingface.co/jorge-erdb)\n- **Importance matrix**: [bartowski](https://huggingface.co/bartowski) — imatrix calibration dataset\n- **Fine-tune**: [Jackrong](https://huggingface.co/Jackrong) — Claude 4.6 Opus reasoning distillation via Unsloth + LoRA\n- **Base model**: [Qwen Team](https://huggingface.co/Qwen) — Qwen3.5-35B-A3B\n\n---\n\n# 🌟 Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled\n\n🔥 **Update (April 5):** I’ve released the complete training notebook, codebase, and a comprehensive PDF guide to help beginners and enthusiasts understand and reproduce this model's fine-tuning process. \n\n> ❤️ Special thanks to the [**Unsloth**](https://unsloth.ai) open-source library and [@KyleHessling1](https://x.com/kylehessling1) for their support.\n\n## 📚 Resources & Guides\n\n👉 **[GitHub Repository: Jackrong-llm-finetuning-guide](https://github.com/R6410418/Jackrong-llm-finetuning-guide.git)**\nVisit the repo to dive into the codebase and reproduce the results locally or on Colab.\n\n### 📥 Core Technical Document\n**🔗 [Qwopus3.5-27b Complete Fine-Tuning Guide (PDF)](https://github.com/R6410418/Jackrong-llm-finetuning-guide/blob/main/guidePDF/Qwopus3-5-27b-Colab_complete_guide_to_llm_finetuning.pdf)**\n* **The Full Pipeline:** A step-by-step walkthrough—from downloading the base model and unifying heterogeneous data, to configuring trainer hyperparameters and publishing to Hugging Face.\n* **Beginner Friendly:** Includes an introductory guide to getting started with Google Colab and Unsloth.\n* *Feedback welcome! If you spot any areas for improvement, please let me know and I will update it promptly.*\n\n> **A Note:**\n> My goal isn't just to detail a workflow, but to demystify LLM training. Beyond the social media hype, fine-tuning isn't an unattainable ritual—often, all you need is a Google account, a standard laptop, and relentless curiosity. \n> \n> *No one starts as an expert, but every expert was once brave enough to begin.*\n> \n> All training and testing for this project were self-funded. If you find this model or guide helpful, a **Star ⭐️ on GitHub** would be the greatest encouragement. Thank you! 🙏\n\n> [!Note]\n> The Claude series model optimizations are named under the **Qwopus3.5 series**, with the latest version being **🌟Qwopus3.5-v3**.\n\n---\n\n![HB8AleUaMAArNyM](https://cdn-uploads.huggingface.co/production/uploads/66309bd090589b7c65950665/GHkMJL6I383eIwK1qj80K.jpeg)\n\n\n## 💡 Model Introduction\n**Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled** is a highly capable reasoning model fine-tuned on top of the powerful Qwen3.5 architecture. The model's core directive is to leverage state-of-the-art Chain-of-Thought (CoT) distillation primarily sourced from Claude-4.6 Opus interactions. \n\nThrough Supervised Fine-Tuning (SFT) focusing specifically on structured reasoning logic, this model excels in breaking down complex user problems, planning step-by-step methodologies within strictly formatted `<think>` tags, and ultimately delivering precise, nuanced solutions. \n\n### 🧠 Example of Learned Reasoning Scaffold（Example）\n\nThe model includes targeted optimizations addressing Qwen3.5’s tendency toward excessive transitional or repetitive reasoning on simple queries. Through deep distillation and structural imitation of Claude-4.6-Opus reasoning chains, the model adopts a more efficient structured thinking pattern:  \n**“Let me analyze this request carefully: 1..2..3...”.**  \nThis streamlined reasoning paradigm significantly reduces redundant cognitive loops while preserving deep analytical capacity, resulting in substantially improved inference efficiency.\n\n```text\nLet me analyze this request carefully:\n\n1. Identify the core objective of the problem.\n2. Break the task into clearly defined subcomponents.\n3. Evaluate constraints and edge cases.\n4. Formulate a step-by-step solution plan.\n5. Execute the reasoning sequentially and verify consistency.\n            .\n            .\n            .\n```\n\n## 🗺️ Training Pipeline Overview\n\n```text\nBase Model (Qwen3.5-35B-A3B)\n │\n ▼\nSupervised Fine-Tuning (SFT) + LoRA\n │\n ▼\nFinal Model (Claude-4.6-Opus-Reasoning-Distilled,text-only)\n```\n\n## 📋 Stage Details\n\n### 🔹 Supervised Fine-Tuning (SFT)\n- **Objective:** To inject high-density reasoning logic and establish a strict format for problem-solving involving an internal thinking state prior to outputting the final response.\n- **Methodology:** We utilized **Unsloth** for highly efficient memory and compute optimization. A critical component of this stage is the `train_on_responses_only` strategy, masking instructions so the loss is purely calculated over the generation of the `<think>` sequences and the subsequent solutions. \n- **Format Enforcement:** All training samples were systematically normalized so the model strictly abides by the structure `<think> {internal reasoning} </think>\\n {final answer}`.\n\n### 📚 All Datasets Used\nThe dataset consists of high-quality, filtered reasoning distillation data:\n\n| Dataset Name | Description / Purpose |\n|--------------|-----------------------|\n| [nohurry/Opus-4.6-Reasoning-3000x-filtered](https://huggingface.co/datasets/nohurry/Opus-4.6-Reasoning-3000x-filtered) | Provides comprehensive Claude 4.6 Opus reasoning trajectories. |\n| [TeichAI/claude-4.5-opus-high-reasoning-250x](https://huggingface.co/datasets/TeichAI/claude-4.5-opus-high-reasoning-250x) | Injecting high-intensity, structured reasoning instances. |\n| [Jackrong/Qwen3.5-reasoning-700x](https://huggingface.co/datasets/Jackrong/Qwen3.5-reasoning-700x) | Additional curated reasoning samples designed to strengthen structured step-by-step problem solving and improve reasoning diversity. |\n\n## 🌟 Core Skills & Capabilities\n1. **Modular & Structured Thinking:** Inheriting traits from Opus-level reasoning, the model demonstrates confident parsing of the prompt, establishing an outlined plan in its `<think>` block sequentially rather than exploratory \"trial-and-error\" self-doubt.\n2. **Extended Context Support:** Fine-tuned smoothly with an 8192 context window allowing complex multi-step reasoning traces to exist gracefully within memory limits.\n\n## ⚠️ Limitations & Intended Use\n- **Hallucination Risk:** While reasoning is strong, the model remains an autoregressive LLM; external facts provided during the thinking sequence may occasionally contain hallucinations if verifying real-world events.\n- **Intended Scenario:** Best suited for offline analytical tasks, coding, math, and heavy logic-dependent prompting where the user needs to transparently follow the AI's internal logic.\n- **Preview Version Notice:** Because this model is relatively new and intentionally lightweight, the surrounding ecosystem — including inference templates, fine-tuning pipelines, routing configurations, and tooling integrations — may not yet be fully mature or standardized. As a result, users may encounter occasional bugs, compatibility inconsistencies, or integration edge cases. The current release should be considered a preview build while the broader architectural stack and supporting utilities continue to stabilize and improve.\n\n### ⚠️ Training Disclaimer\n\nDuring the fine-tuning process, the Triton kernel required approximately **131072 bytes of shared memory per CUDA block**. On some GPUs this exceeded the available shared memory limits, which caused kernel execution issues. To ensure training stability and proper kernel execution, the fine-tuning was therefore conducted on **80GB VRAM GPUs**.\n\nThis model was fine-tuned using a **LoRA-based parameter-efficient training strategy**, where only a small subset of parameters were updated. In total, **465,551,360 parameters were trainable out of 35,572,733,296 total parameters**, corresponding to **approximately 1.31% of the model being trained**.\n\nDuring training, the loss curve exhibited noticeable fluctuations, which is common in LoRA-based reasoning distillation tasks. However, the overall trend remained **consistently decreasing**, with the training loss eventually converging to approximately **0.384**.\n## 🙏 Acknowledgements\nSignificant thanks to the [Unsloth AI](https://unsloth.ai/) team for making rapid fine-tuning of MoE and large LLM models accessible. Additionally, we acknowledge Qwen internally, and the open-source community developers producing exceptional distilled datasets (`nohurry` and `TeichAI`).\n\n## 📖 Citation\n\nIf you use this model in your research or projects, please cite:\n\n```bibtex\n@misc{jackrong_qwen35_opus_distilled,\n  title        = {Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled},\n  author       = {Jackrong},\n  year         = {2026},\n  publisher    = {Hugging Face},\n  howpublished = {\\url{https://huggingface.co/Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled}}\n}\n```",
    "related_quantizations": []
  },
  "tags": [
    "transformers",
    "gguf",
    "text-generation-inference",
    "unsloth",
    "qwen3_5_moe",
    "qwen",
    "qwen3.5",
    "reasoning",
    "chain-of-thought",
    "4bit",
    "imatrix",
    "image-text-to-text",
    "zh",
    "en",
    "ko",
    "dataset:nohurry/Opus-4.6-Reasoning-3000x-filtered",
    "dataset:Jackrong/Qwen3.5-reasoning-700x",
    "base_model:Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled",
    "base_model:quantized:Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 912,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-08T23:13:26.000Z",
  "created_at": "2026-04-08T16:55:54.000Z",
  "pipeline_tag": "image-text-to-text",
  "library_name": "transformers"
}

{
  "_id": "69d6889a7e0be2332a776643",
  "id": "jorge-erdb/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-4bit-GGUF",
  "modelId": "jorge-erdb/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-4bit-GGUF",
  "sha": "790fdc929fd228110b137032ec6bc25e0fd7eb4a",
  "createdAt": "2026-04-08T16:55:54.000Z",
  "lastModified": "2026-04-08T23:13:26.000Z",
  "author": "jorge-erdb",
  "downloads": 912,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "image-text-to-text",
  "library_name": "transformers",
  "siblings_count": 4
}