richarderkhov/lingzhi-ai_-_lingzhi-0.5b-chat-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "🌐 官方网站，欢迎访问",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nLingzhi-0.5B-chat - GGUF\n- Model creator: https://huggingface.co/Lingzhi-AI/\n- Original model: https://huggingface.co/Lingzhi-AI/Lingzhi-0.5B-chat/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Lingzhi-0.5B-chat.Q2_K.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q2_K.gguf) | Q2_K | 0.32GB |\n| [Lingzhi-0.5B-chat.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.IQ3_XS.gguf) | IQ3_XS | 0.32GB |\n| [Lingzhi-0.5B-chat.IQ3_S.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.IQ3_S.gguf) | IQ3_S | 0.32GB |\n| [Lingzhi-0.5B-chat.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q3_K_S.gguf) | Q3_K_S | 0.32GB |\n| [Lingzhi-0.5B-chat.IQ3_M.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.IQ3_M.gguf) | IQ3_M | 0.32GB |\n| [Lingzhi-0.5B-chat.Q3_K.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q3_K.gguf) | Q3_K | 0.33GB |\n| [Lingzhi-0.5B-chat.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q3_K_M.gguf) | Q3_K_M | 0.33GB |\n| [Lingzhi-0.5B-chat.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q3_K_L.gguf) | Q3_K_L | 0.34GB |\n| [Lingzhi-0.5B-chat.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.IQ4_XS.gguf) | IQ4_XS | 0.33GB |\n| [Lingzhi-0.5B-chat.Q4_0.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q4_0.gguf) | Q4_0 | 0.33GB |\n| [Lingzhi-0.5B-chat.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.IQ4_NL.gguf) | IQ4_NL | 0.33GB |\n| [Lingzhi-0.5B-chat.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q4_K_S.gguf) | Q4_K_S | 0.36GB |\n| [Lingzhi-0.5B-chat.Q4_K.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q4_K.gguf) | Q4_K | 0.37GB |\n| [Lingzhi-0.5B-chat.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q4_K_M.gguf) | Q4_K_M | 0.37GB |\n| [Lingzhi-0.5B-chat.Q4_1.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q4_1.gguf) | Q4_1 | 0.35GB |\n| [Lingzhi-0.5B-chat.Q5_0.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q5_0.gguf) | Q5_0 | 0.37GB |\n| [Lingzhi-0.5B-chat.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q5_K_S.gguf) | Q5_K_S | 0.38GB |\n| [Lingzhi-0.5B-chat.Q5_K.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q5_K.gguf) | Q5_K | 0.39GB |\n| [Lingzhi-0.5B-chat.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q5_K_M.gguf) | Q5_K_M | 0.39GB |\n| [Lingzhi-0.5B-chat.Q5_1.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q5_1.gguf) | Q5_1 | 0.39GB |\n| [Lingzhi-0.5B-chat.Q6_K.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q6_K.gguf) | Q6_K | 0.47GB |\n| [Lingzhi-0.5B-chat.Q8_0.gguf](https://huggingface.co/RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf/blob/main/Lingzhi-0.5B-chat.Q8_0.gguf) | Q8_0 | 0.49GB |\n\n\n\n\nOriginal model description:\n# 灵智大模型 - 垂直领域行业专家\n\n🌐 [官方网站，欢迎访问](https://ailingzhi.com)\n\n## ✨ 亮点\n- 从Qwen2-base完美复现了Qwen2-chat，并公开了训练数据；\n- 在垂类领域训练场景下，灵智模型能够在提升垂类领域性能的同时也保持了通用领域的性能；\n- 对多种训练范式（例如直接指令微调，先持续预训练再指令微调等八种范式）做了总结，并针对不同的模型大小采取了最佳的训练范式；\n- 开源了8个灵智模型：`Lingzhi-0.5B-chat`, `Lingzhi-0.8B-chat`, `Lingzhi-1.5B-chat`, `Lingzhi-2.7B-chat`, `Lingzhi-7B-chat`, `Lingzhi-10B-chat`, `Lingzhi-57MOE14B-chat`, `Lingzhi-72B-chat`.\n\n## 📄 摘要\n在实际应用中，当预训练数据不可用时，进行**持续训练**是很常见的。然而，持续训练往往会在增强领域特定技能的同时导致大语言模型（LLMs）灾难性地遗忘其通用能力。在本文中，我们首先对常见的持续训练范式进行了实证研究，然后选择了最佳范式来训练灵智系列模型。实验表明，灵智能够在保持通用能力的同时增强领域特定的性能。我们已经开源了所有模型、训练数据和基准测试，用户可以将它们应用到自己的领域特定区域。\n\n## 📘 介绍\n大语言模型（LLMs）近年来因其在各种实际下游任务中的出色表现而备受关注。实际上，尽管现有的LLMs在通用领域表现良好，但由于在预训练或指令微调期间缺乏特定领域的专业暴露，它们可能在用户需要的特定领域（如会计、法律、金融）中表现不佳。\n\n为了提升LLMs在特定领域的表现，我们需要收集相应的数据进行持续学习，如持续预训练（CPT）或有监督微调（SFT）。然而，我们注意到，仅在特定领域进行持续学习可能导致通用能力的灾难性遗忘，如规划、指令执行、数学、编程和自然语言理解等。\n\n为了同时保持通用和领域特定能力，通常会部署一个未修改的原生模型用于通用任务，而一个微调模型用于专业任务。这将对计算硬件资源（如GPU和内存）提出巨大的需求，从而阻碍商业部署。众所周知，上述现象是业界面临的一个非常棘手的问题。因此，一个值得研究的问题出现了：如何在持续学习过程中提高领域特定的表现，而不损害通用能力？\n\n为了解决这个问题，我们进行了实证研究，探索了各种持续学习范式并总结了它们的优缺点。最终，在实证研究之后，我们选择了最佳的学习范式和训练数据，基于Qwen2-base进行持续学习，衍生出我们的灵智系列模型。经过大量实验，灵智能够在多个特定领域中表现出色，同时在通用能力方面也表现出与原始Qwen2-chat模型相当的性能。\n\n## 📋 示例\n1. huggingface示例代码\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nlingzhi_model_path = \"Lingzhi-AI/Lingzhi-7B-chat\"\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    lingzhi_model_path,\n    torch_dtype=\"auto\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(lingzhi_model_path)\n\nprompt = \"帮我介绍一下灵智大模型。\"\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": prompt}\n]\ntext = tokenizer.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True\n)\nmodel_inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n\ngenerated_ids = model.generate(\n    model_inputs.input_ids,\n    max_new_tokens=512\n)\ngenerated_ids = [\n    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)\n]\n\nresponse = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\nprint(response)\n```\n\n2. modelscope示例代码\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nlingzhi_model_path = \"LingzhiLLM/Lingzhi-7B-chat\"\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    lingzhi_model_path,\n    torch_dtype=\"auto\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(lingzhi_model_path)\n\nprompt = \"帮我介绍一下灵智大模型。\"\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": prompt}\n]\ntext = tokenizer.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True\n)\nmodel_inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n\ngenerated_ids = model.generate(\n    model_inputs.input_ids,\n    max_new_tokens=512\n)\ngenerated_ids = [\n    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)\n]\n\nresponse = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\nprint(response)\n```\n\n## 📊 结果\n\n> 备注：Baselines中Qwen2的所有结果均是在我们统一的环境下进行评测的。\n\n| **Base Model**       | **General** |        |             |        |          |        |           |        | **Domains** |        | **Avg.** |\n| :-------------------- | :---------- | :----- | :---------- | :----- | :------- | :----- | :-------- | :----- | :---------- | :----- | :------- |\n|                       | **English** |        | **Chinese** |        | **Math** |        | **Code**  |        |             |        |          |\n|                       | MMLU        | BBH    | C-Eval      | CMMLU  | GSM8K    | MathQA | HumanEval | MBPP   | Account     | Law    |          |\n| ***Baselines***       |             |        |             |        |          |        |           |        |             |        |          |\n| Qwen2-0\\.5B-chat      | 43\\.30      | 10\\.35 | 54\\.16      | 53\\.57 | 33\\.97   | 25\\.76 | 20\\.73    | 12\\.40 | 17\\.01      | 25\\.00 | 29\\.62   |\n| Qwen2-1\\.5B-chat      | 55\\.73      | 9\\.55  | 69\\.32      | 70\\.13 | 54\\.21   | 32\\.93 | 42\\.68    | 20\\.60 | 32\\.65      | 42\\.07 | 42\\.99   |\n| Qwen2-7B-chat         | 69\\.82      | 30\\.56 | 81\\.58      | 81\\.77 | 66\\.26   | 44\\.09 | 72\\.56    | 42\\.20 | 55\\.10      | 59\\.15 | 60\\.31   |\n| Qwen2-57MOE14B-chat   |             |        |             |        |          |        |           |        |             |        |          |\n| Qwen2-72B-chat        |             |        |             |        |          |        |           |        |             |        |          |\n| ***Lingzhi Models***  |             |        |             |        |          |        |           |        |             |        |          |\n| Lingzhi-0\\.5B-chat    | 44\\.25      | 25\\.65 | 55\\.05      | 53\\.74 | 29\\.34   | 29\\.18 | 25\\.00    | 22\\.40 | 25\\.85      | 40\\.24 | 35\\.07   |\n| Lingzhi-0\\.8B-chat    | 42\\.93      | 27\\.77 | 53\\.34      | 50\\.98 | 21\\.00   | 28\\.84 | 28\\.66    | 18\\.00 | 24\\.49      | 40\\.85 | 33\\.69   |\n| Lingzhi-1\\.5B-chat    | 55\\.35      | 33\\.67 | 69\\.47      | 69\\.10 | 49\\.58   | 35\\.31 | 39\\.02    | 31\\.00 | 37\\.41      | 42\\.68 | 46\\.26   |\n| Lingzhi-2\\.7B-chat    | 53\\.65      | 36\\.77 | 67\\.09      | 67\\.39 | 46\\.02   | 34\\.51 | 40\\.85    | 30\\.00 | 38\\.10      | 60\\.98 | 47\\.54   |\n| Lingzhi-7B-chat       | 69\\.06      | 58\\.95 | 82\\.69      | 83\\.05 | 74\\.22   | 45\\.59 | 56\\.10    | 49\\.80 | 72\\.79      | 89\\.02 | 68\\.13   |\n| Lingzhi-10B-chat      | 69\\.37      | 64\\.37 | 81\\.50      | 82\\.27 | 76\\.19   | 46\\.00 | 60\\.98    | 50\\.40 | 70\\.07      | 82\\.93 | 68\\.41   |\n| Lingzhi-57MOE14B-chat |             |        |             |        |          |        |           |        |             |        |          |\n| Lingzhi-72B-chat      |             |        |             |        |          |        |           |        |             |        |          |\n\n\n## 📚 引用\n<span style=\"color:orange;\">⚠️ **警告**</span> 如果您用到了我们的模型和数据，请使用以下参考文献。\n```\n@misc{lingzhi,\n      title={Lingzhi: Improving Domain-Specific Performance without Compromising General Capabilities}, \n      author={Daoguang Zan, Lei Yu, Ailun Yu, Zhirong Huang, Zongshuai Ruan, Pengjie Huang},\n      year={2024},\n      note={All authors contributed equally. The computational power required to train the Lingzhi models (12*8 H800 80G) was provided by Lingzhi AI. Special thanks to them.}\n}\n```\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 216,
  "gated": false,
  "private": false,
  "last_modified": "2024-07-22T12:47:37.000Z",
  "created_at": "2024-07-22T12:37:51.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "669e529f6582d2ef7084463f",
  "id": "RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf",
  "modelId": "RichardErkhov/Lingzhi-AI_-_Lingzhi-0.5B-chat-gguf",
  "sha": "43a737e5fc0c5e89fd91be9e977bb3d0d3681988",
  "createdAt": "2024-07-22T12:37:51.000Z",
  "lastModified": "2024-07-22T12:47:37.000Z",
  "author": "RichardErkhov",
  "downloads": 216,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}