angelslim/hy-1.8b-2bit-gguf overview

{
  "metadata": {},
  "card_data": {
    "tags": [
      "hy",
      "quant",
      "2bit"
    ],
    "base_model": [
      "AngelSlim/HY-1.8B-2Bit"
    ],
    "frontmatter": {
      "tags": [
        "hy",
        "quant",
        "2bit"
      ],
      "base_model": [
        "AngelSlim/HY-1.8B-2Bit"
      ]
    },
    "hero_image_url": "https://github.com/Tencent/AngelSlim/blob/main/docs/source/assets/logos/angelslim_logo.png?raw=true",
    "summary": "./bin/llama-cli -m hunyuan-q2_0.gguf -p \"写一副春联\" -t 1 --seed 4568 -n 32 # no thinking ./bin/llama-cli -m hunyuan-q2_0.gguf -p \"/no_think写一副春联\" -t 1 --seed 4568 -n 32 `` #### Run the llama.cpp benchmark The general command is: `bash ./bin/llama-bench -m hunyuan-q2_0.gguf -p  -t  -n  `` !image/jpeg",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\ntags:\n- hy\n- quant\n- 2bit\nbase_model:\n- AngelSlim/HY-1.8B-2Bit\n---\n\n<p align=\"center\">\n  <picture>\n    <source media=\"(prefers-color-scheme: dark)\" srcset=\"https://github.com/Tencent/AngelSlim/blob/main/docs/source/assets/logos/angelslim_logo_light.png?raw=true\">\n    <img alt=\"AngelSlim\" src=\"https://github.com/Tencent/AngelSlim/blob/main/docs/source/assets/logos/angelslim_logo.png?raw=true\" width=55%>\n  </picture>\n</p>\n\n<h3 align=\"center\">\nDedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit.\n</h3>\n\n<p align=\"center\">\n          📣 <a href=\"https://huggingface.co/AngelSlim/HY-1.8B-2Bit-GGUF\">GGUF</a>&nbsp&nbsp | ✒️ <a href=\"http://arxiv.org/abs/2602.21233\">TechnicalReport</a>&nbsp&nbsp | &nbsp&nbsp 📖 <a href=\"https://angelslim.readthedocs.io/\">Documentation</a>&nbsp&nbsp | &nbsp&nbsp🤗 <a href=\"https://huggingface.co/AngelSlim\">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp🤖 <a href=\"https://modelscope.cn/organization/AngelSlim\">ModelScope</a>&nbsp&nbsp | &nbsp&nbsp💬 <a href=\"./docs/source/assets/angel_slim_wechat.png\">WeChat</a>\n<br>\n</p>\n\n![image/jpeg](2bit-benchmark.png)\n\n## 📣Latest News\n- [26/02/09] We have released HY-1.8B-2Bit, 2bit on-device large language model.\n- [26/01/13] We have released v0.3. We support the training and deployment of Eagle3 for all-scale LLMs/VLMs/Audio models, as detailed in the [guidance documentation](https://angelslim.readthedocs.io/zh-cn/latest/features/speculative_decoding/eagle/index.html). And We released **Sherry**, the hardware-efficient 1.25 bit quantization algorithm [Paper Comming soon] | [[Code]](https://github.com/Tencent/AngelSlim/tree/sherry/Sherry)🔥🔥🔥\n\nFor more detailed information, please refer to[[AngelSlim]](https://github.com/Tencent/AngelSlim)\n\n## 🌟HY-1.8B-2Bit Key Features\n\n- **Superior Model Capability** HY-1.8B-2Bit is developed via Quantization-Aware Training (QAT) based on the Hunyuan-1.8B-Instruct backbone. By aggressively compressing the model to a 2-bit weight precision, we achieve a performance profile that remains highly competitive with PTQ-INT4 benchmarks. Across a multi-dimensional evaluation suite—encompassing mathematics, humanities, and programming—HY-1.8B-2Bit exhibits a marginal performance degradation of only 4\\% compared to its full-precision counterpart, demonstrating exceptional information retention despite the radical reduction in bit-width. \n\n- **Unmatched Scale-to-Performance Efficiency** When compared to dense models of equivalent size (e.g., 0.5B parameters), HY-1.8B-2Bit demonstrates a substantial competitive advantage, outperforming benchmarks by an average of 16\\% across core competencies. As a state-of-the-art (SOTA) solution for its parameter class, HY-1.8B-2Bit provides an extensible and highly efficient alternative for edge computing, delivering high-tier reasoning capabilities within a compact footprint. \n\n- **Comprehensive Reasoning Proficiency** HY-1.8B-2Bit inherits the complete \"full-thinking\" capabilities of the Hunyuan-1.8B-Instruct model, marking it as the industry's most compact model to support sophisticated reasoning pathways. By integrating a Dual Chain-of-Thought (Dual-CoT) strategy, the model empowers users to navigate the trade-off between latency and depth: utilizing concise short-CoT for intuitive queries and detailed long-CoT for computationally intensive tasks. This flexibility ensures that HY-1.8B-2Bit can be seamlessly deployed in real-time, resource-constrained environments that demand both rapid response and high-fidelity logical synthesis.\n\n\n## 📈 Benchmark\n\nBenchmark results for HY-1.8B-2Bit equivalent weights on vLLM across **cmmlu**,**ceval**,**arc**,**bbh**,**gsm8k**,**humaneval**,**livecodebench** and **gpqa_diamond**.\n\nThe empirical results reveal that HY-1.8B-2Bit maintains high-tier performance despite the extreme reduction in bit-width, incurring a marginal average degradation of only 3.97\\% compared to its full-precision 1.8B teacher. Remarkably, HY-1.8B-2Bit performs nearly on par with the INT4 variant,with a negligible accuracy gap of only 0.13\\%, while utilizing only half the weight precision. When compared to the dense HY-0.5B model, which occupies a comparable model size, the superiority of the 2-bit QAT approach becomes evident.  While the 0.5B dense model suffers a catastrophic 21.87\\% drop in average accuracy, HY-1.8B-2Bit remains robust, outperforming the smaller dense counterpart by 22.29\\% in GSM8K and 20.62\\% in LiveCodeBench. \n\n| Model            | cmmlu  | ceval  | arc    | bbh    | gsm8k  | humaneval<br/>(pass@3) | livecodebench | gpqa_diamond<br/>(pass@3) |\n|------------------|--------|--------|--------|--------|--------|-------------------|---------------|----------------------|\n| HY-1.8B          | 55.07% | 54.27% | 70.50% | 79.08% | 84.08% | 94.51%            | 31.50%        | 68.18%               |\n| HY-0.5B          | 37.08% | 35.98% | 49.89% | 58.10% | 55.04% | 67.07%            | 12.11%        | 46.97%               |\n| HY-1.8B-int4gptq | 50.80% | 48.67% | 68.83% | 74.80% | 78.70% | 89.02%            | 30.08%        | 65.56%               |\n| **HY-1.8B-2Bit**      | 49.32% | 47.60% | 64.45% | 75.54% | 77.33% | 93.29%            | 32.73%        | 65.15%               |\n\n\n\n## 💻Deployment \nThis setup ONLY works on SME2-capable devices (for example, Apple M4, vivo x300 and Arm CPUs with SME2 support). Neon kernel will follow up.\n\n### Running Hunyuan model on MacBook M4\n\nWe have provided the converted GGUF file, [[LINK]](https://huggingface.co/AngelSlim/HY-1.8B-2Bit-GGUF)\n\n\n Clone llama.cpp\n\n```bash\ngit clone https://github.com/ggml-org/llama.cpp.git\n```\n\nEnter the llama.cpp folder\n\n```bash\ncd llama.cpp\n```\n\nFetch and check out the PR branch\n```bash\ngit fetch origin pull/19357/head:pr-19357-sme2-int2\ngit checkout pr-19357-sme2-int2\n```\n\nBuild llama.cpp with KleidiAI enabled\n\n```bash\nmkdir build && cd build\n\ncmake -DGGML_CPU_KLEIDIAI=ON -DGGML_METAL=OFF -DGGML_BLAS=OFF ..\n\nmake -j8\n```\n\nQuantize the Hunyuan fp16 model to int2 per-channel (q2_0c)\n\n```bash\n./bin/llama-quantize hunyuan-fp16-qdq.gguf hunyuan-q2_0.gguf q2_0c\n```\n\n#### Run the CLI llama.cpp example\n\n\n```bash\nexport GGML_KLEIDIAI_SME=1\n\n# thinking \n./bin/llama-cli -m hunyuan-q2_0.gguf -p \"写一副春联\" -t 1 --seed 4568 -n 32\n# no thinking \n./bin/llama-cli -m hunyuan-q2_0.gguf -p \"/no_think写一副春联\" -t 1 --seed 4568 -n 32\n```\n\n\n\n#### Run the llama.cpp benchmark\n\nThe general command is:\n\n```bash\n./bin/llama-bench -m hunyuan-q2_0.gguf -p <prompt-length> -t <number-of-threads> -n <gen-length>\n```\n\n![image/jpeg](m4_performance.png)\n\n\n## 📝 License\n\nThe code for this project is open-sourced under the [License for AngelSlim](LICENSE).\n\n## 🔗 Citation\n\n```\n@article{angelslim2026,\n  title={AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression},\n  author={Hunyuan AI Infra Team},\n  journal={arXiv preprint arXiv:2602.21233},\n  year={2026}\n}\n```\n\n## 💬 Technical Discussion\n\n* AngelSlim is continuously iterating and new features will be released soon. If you have any questions or suggestions, please open an issue on [GitHub Issues](https://github.com/Tencent/AngelSlim/issues) or join our [WeChat discussion group](https://github.com/Tencent/AngelSlim/blob/main/docs/source/assets/angel_slim_wechat.png?raw=true).",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "hunyuan_v1_dense",
    "hy",
    "quant",
    "2bit",
    "arxiv:2602.21233",
    "base_model:AngelSlim/HY-1.8B-2Bit",
    "base_model:quantized:AngelSlim/HY-1.8B-2Bit",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 39,
  "downloads": 523,
  "gated": false,
  "private": false,
  "last_modified": "2026-03-18T12:26:22.000Z",
  "created_at": "2026-02-04T03:26:53.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "6982bc7de346e0f1b501de57",
  "id": "AngelSlim/HY-1.8B-2Bit-GGUF",
  "modelId": "AngelSlim/HY-1.8B-2Bit-GGUF",
  "sha": "09963dec1f2ead46c0245e135427a0d1f0c8450b",
  "createdAt": "2026-02-04T03:26:53.000Z",
  "lastModified": "2026-03-18T12:26:22.000Z",
  "author": "AngelSlim",
  "downloads": 523,
  "likes": 39,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 17
}