richarderkhov/fuseai_-_fusechat-7b-varm-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "./assets/fig_0.png",
    "summary": "tokens = tokenizer(\"GPT4 Correct User: HelloGPT4 Correct Assistant:\").input_ids assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747] # Multi-turn tokens = tokenizer(\"GPT4 Correct User: HelloGPT4 Correct Assistant: HiGPT4 Correct User: How are you today?GPT4 Correct Assistant:\").input_ids assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747] `` The GPT4 template is also available as the integrated tokenizer.chat_template, which can be used instead of manually specifying the template: `python messages = [ {\"role\": \"user\", \"content\": \"Hello\"}, {\"role\": \"assistant\", \"content\": \"Hi\"}, {\"role\": \"user\", \"content\": \"How are you today?\"} ] tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True) assert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747] ``",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nFuseChat-7B-VaRM - GGUF\n- Model creator: https://huggingface.co/FuseAI/\n- Original model: https://huggingface.co/FuseAI/FuseChat-7B-VaRM/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [FuseChat-7B-VaRM.Q2_K.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q2_K.gguf) | Q2_K | 2.53GB |\n| [FuseChat-7B-VaRM.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.IQ3_XS.gguf) | IQ3_XS | 2.81GB |\n| [FuseChat-7B-VaRM.IQ3_S.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.IQ3_S.gguf) | IQ3_S | 2.96GB |\n| [FuseChat-7B-VaRM.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q3_K_S.gguf) | Q3_K_S | 2.95GB |\n| [FuseChat-7B-VaRM.IQ3_M.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.IQ3_M.gguf) | IQ3_M | 3.06GB |\n| [FuseChat-7B-VaRM.Q3_K.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q3_K.gguf) | Q3_K | 3.28GB |\n| [FuseChat-7B-VaRM.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q3_K_M.gguf) | Q3_K_M | 3.28GB |\n| [FuseChat-7B-VaRM.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q3_K_L.gguf) | Q3_K_L | 3.56GB |\n| [FuseChat-7B-VaRM.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.IQ4_XS.gguf) | IQ4_XS | 3.67GB |\n| [FuseChat-7B-VaRM.Q4_0.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q4_0.gguf) | Q4_0 | 3.83GB |\n| [FuseChat-7B-VaRM.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.IQ4_NL.gguf) | IQ4_NL | 3.87GB |\n| [FuseChat-7B-VaRM.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q4_K_S.gguf) | Q4_K_S | 3.86GB |\n| [FuseChat-7B-VaRM.Q4_K.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q4_K.gguf) | Q4_K | 4.07GB |\n| [FuseChat-7B-VaRM.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q4_K_M.gguf) | Q4_K_M | 4.07GB |\n| [FuseChat-7B-VaRM.Q4_1.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q4_1.gguf) | Q4_1 | 4.24GB |\n| [FuseChat-7B-VaRM.Q5_0.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q5_0.gguf) | Q5_0 | 4.65GB |\n| [FuseChat-7B-VaRM.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q5_K_S.gguf) | Q5_K_S | 4.65GB |\n| [FuseChat-7B-VaRM.Q5_K.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q5_K.gguf) | Q5_K | 4.78GB |\n| [FuseChat-7B-VaRM.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q5_K_M.gguf) | Q5_K_M | 4.78GB |\n| [FuseChat-7B-VaRM.Q5_1.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q5_1.gguf) | Q5_1 | 5.07GB |\n| [FuseChat-7B-VaRM.Q6_K.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q6_K.gguf) | Q6_K | 5.53GB |\n| [FuseChat-7B-VaRM.Q8_0.gguf](https://huggingface.co/RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf/blob/main/FuseChat-7B-VaRM.Q8_0.gguf) | Q8_0 | 7.17GB |\n\n\n\n\nOriginal model description:\n---\nlicense: apache-2.0\nlanguage:\n- en\nbase_model: openchat/openchat_3.5\ndatasets:\n- FuseAI/FuseChat-Mixture\npipeline_tag: text-generation\ntags:\n- mistral\n- mixtral\n- solar\n- model-fusion\n- fusechat\nlibrary_name: transformers\nmodel-index:\n- name: FuseChat-7B-VaRM\n  results:\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: MT-Bench\n      type: unknown\n    metrics:\n    - type: unknown\n      value: 8.22\n      name: score\n    source:\n      url: https://huggingface.co/spaces/lmsys/mt-bench\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: AI2 Reasoning Challenge (25-Shot)\n      type: ai2_arc\n      config: ARC-Challenge\n      split: test\n      args:\n        num_few_shot: 25\n    metrics:\n    - type: acc_norm\n      value: 62.88\n      name: normalized accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/FuseChat-7B-VaRM\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: HellaSwag (10-Shot)\n      type: hellaswag\n      split: validation\n      args:\n        num_few_shot: 10\n    metrics:\n    - type: acc_norm\n      value: 84.25\n      name: normalized accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/FuseChat-7B-VaRM\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: MMLU (5-Shot)\n      type: cais/mmlu\n      config: all\n      split: test\n      args:\n        num_few_shot: 5\n    metrics:\n    - type: acc\n      value: 63.71\n      name: accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/FuseChat-7B-VaRM\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: TruthfulQA (0-shot)\n      type: truthful_qa\n      config: multiple_choice\n      split: validation\n      args:\n        num_few_shot: 0\n    metrics:\n    - type: mc2\n      value: 45.67\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/FuseChat-7B-VaRM\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: Winogrande (5-shot)\n      type: winogrande\n      config: winogrande_xl\n      split: validation\n      args:\n        num_few_shot: 5\n    metrics:\n    - type: acc\n      value: 79.16\n      name: accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/FuseChat-7B-VaRM\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: GSM8k (5-shot)\n      type: gsm8k\n      config: main\n      split: test\n      args:\n        num_few_shot: 5\n    metrics:\n    - type: acc\n      value: 63.46\n      name: accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FuseAI/FuseChat-7B-VaRM\n      name: Open LLM Leaderboard\n---\n<p align=\"center\" width=\"100%\">\n</p>\n\n<div id=\"top\" align=\"center\">\n\n<p style=\"font-size: 30px; font-weight: bold;\">FuseChat: Knowledge Fusion of Chat Models</p>\n\n<p style=\"font-size: 24px; font-weight: bold;\">[SOTA 7B LLM on MT-Bench]</p>\n\n<h4> |<a href=\"https://arxiv.org/abs/2402.16107\"> 📑 Paper </a> |\n<a href=\"https://huggingface.co/FuseAI\"> 🤗 HuggingFace Repo </a> |\n<a href=\"https://github.com/fanqiwan/FuseLLM\"> 🐱 GitHub Repo </a> |\n</h4>\n\n<!-- **Authors:** -->\n\n_**Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, Wei Bi**_\n\n\n<!-- **Affiliations:** -->\n\n\n_Sun Yat-sen University_\n\n<p align=\"center\">\n    <img src=\"./assets/fig_0.png\" width=\"70%\"> <br>\n</p>\n\n| Proprietary Models                                                    | #Params | MT-Bench | Open Source Models                                                    | #Params | MT-Bench |\n|-----------------------------------------------------------------------|---------|----------|-----------------------------------------------------------------------|---------|----------|\n| GPT-4-1106-preview                                                    | -       | 9.32     | Qwen1.5-72B-Chat                                                      | 72B     | 8.61     |\n| GPT-4-0613                                                            | -       | 9.18     | Nous-Hermes-2-Mixtral-8x7B-DPO                                        | 8x7B    | 8.33     |\n| GPT-4-0314                                                            | -       | 8.96     | Mixtral-8x7B-Instruct-v0.1                                            | 8x7B    | 8.30     |\n| Mistral Medium                                                        | -       | 8.61     | 🤗 [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM) | 7B      | 8.22     |\n| GPT-3.5-Turbo-0613                                                    | -       | 8.39     | Starling-LM-7B-alpha                                                  | 7B      | 8.09     |\n| GPT-3.5-Turbo-1106                                                    | -       | 8.32     | Tulu-2-DPO-70B                                                        | 70B     | 7.89     |\n| 🤗 [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM) | 7B      | 8.22     | OpenChat-3.5                                                          | 7B      | 7.81     |\n| Claude-2.1                                                            | -       | 8.18     | OpenChat-3.5-0106                                                     | 7B      | 7.80     |\n| Claude-2.0                                                            | -       | 8.06     | WizardLM-70B-v1.0                                                     | 70B     | 7.71     |\n| GPT-3.5-Turbo-0314                                                    | -       | 7.94     | Yi-34B-Chat                                                           | 34B     | 7.67     |\n| Claude-1                                                              | -       | 7.90     | Nous-Hermes-2-SOLAR-10.7B                                             | 10.7B   | 7.66     |\n\n\n</div>\n\n\n## News\n- **Feb 26, 2024:** 🔥🔥 We release [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM), which is the fusion of three prominent chat LLMs with diverse architectures and scales, namely [NH2-Mixtral-8x7B](https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO), [NH2-Solar-10.7B](https://huggingface.co/NousResearch/Nous-Hermes-2-SOLAR-10.7B), and [OpenChat-3.5-7B](https://huggingface.co/openchat/openchat_3.5). FuseChat-7B-VaRM achieves an average performance of **8.22** on MT-Bench, outperforming various powerful chat LLMs at 7B and 34B scales like [Starling-7B](https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha) and [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat), even surpassing [GPT-3.5 (March)](https://platform.openai.com/docs/models/gpt-3-5-turbo), [Claude-2.1](https://www.anthropic.com/news/claude-2-1), and approaching [Mixtral-8x7B-Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1). \n\n- **Feb 25, 2024:** 🔥 We release [FuseChat-Mixture](https://huggingface.co/datasets/FuseAI/FuseChat-Mixture), which is a comprehensive training dataset covers different styles and capabilities, featuring both human-written and model-generated, and spanning general instruction-following and specific skills.\n\n## Contents\n\n- [Overview](#overview)\n- [Model Release](#model-release)\n- [Quick Start](#quick-start)\n- [Data Construction](#data-construction)\n- [Pairwise Knowledge Fusion](#pairwise-knowledge-fusion)\n- [Model Merging](#model-merging)\n- [Evaluation](#evaluation)\n- [Citation](#citation)\n\n## Overview\n \nIn this work, we propose an extended framework of FuseLLM to integrate the collective knowledge and individual strengths of multiple structure and scale-varied chat LLMs into a more powerful chat LLM, resulting in FuseChat. FuseChat adopts a fuse-then-merge strategy with two main stages. Firstly, it undertakes pairwise knowledge fusion for source LLMs to derive multiple target LLMs of identical structure and size via lightweight fine-tuning. Then, these target LLMs are merged within the parameter space, wherein we propose a novel method VaRM for determining the merging weights based on the variation ratio of parameter matrices before and after fine-tuning. \n\n\nMoreover, we argue that the concept of knowledge fusion adopted by both FuseChat and FuseLLM shares a fundamentally similar purpose with other related topics, such as the recently popular topic of mixture of experts (MoEs), because they all aim to leverage the strengths of multiple models (experts). However, while MoEs require loading multiple experts during inference, which has higher memory requirements, knowledge fusion supports the integration of multiple LLMs with diverse architectures into a single LLM without any additional memory requirement, making it more memory-efficient. \n\n<p align=\"center\">\n    <img src=\"./assets/fig_1.png\" width=\"95%\"> <br>\n</p>\n\n\n## Model Release\n\nWe release [FuseChat-7B-VaRM](https://huggingface.co/FuseAI/FuseChat-7B-VaRM), which is the fusion of three prominent chat LLMs with diverse architectures and scales, namely [NH2-Mixtral-8x7B](https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO), [NH2-Solar-10.7B](https://huggingface.co/NousResearch/Nous-Hermes-2-SOLAR-10.7B), and [OpenChat-3.5-7B](https://huggingface.co/openchat/openchat_3.5). FuseChat-7B-VaRM achieves an average performance of **8.22** on MT-Bench, outperforming various powerful chat LLMs at 7B and 34B scales like [Starling-7B](https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha) and [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat), even surpassing [GPT-3.5 (March)](https://platform.openai.com/docs/models/gpt-3-5-turbo), [Claude-2.1](https://www.anthropic.com/news/claude-2-1), and approaching [Mixtral-8x7B-Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1).\n\nTo support a plug-and-play fusion of new source LLM, we release our target LLMs: [OpenChat-3.5-7B-Solar](https://huggingface.co/FuseAI/OpenChat-3.5-7B-Solar) and [OpenChat-3.5-7B-Mixtral](https://huggingface.co/FuseAI/OpenChat-3.5-7B-Mixtral), which are obtained from pair-wise knowledge fusion. Integrating a new source LLM at any scale requires only obtaining a target LLM from the new source LLM and merging it with the existing target LLMs.\n\nWe also release FuseChat with other merging methods: [FuseChat-7B-SLERP](https://huggingface.co/FuseAI/FuseChat-7B-SLERP) and [FuseChat-7B-TA](https://huggingface.co/FuseAI/FuseChat-7B-TA), which achieves an average performance of **8.19** and **8.20** on MT-Bench respectively.\n\nHere are the evaluation results.\n\n<p align=\"center\">\n    <img src=\"./assets/tab_1.png\" width=\"95%\"> <br>\n</p>\n\n## Quick Start\n\n### Setup\n\nWe use `python 3.11` in this project.\n\nThen, we have to install all the libraries listed in `requirements.txt`.\n\n```bash\npip install -r requirements.txt\n```\n\n### Usage\n\nHere's how you can run the model using the 🤗 Transformers:\n\n```python\nimport transformers\ntokenizer = transformers.AutoTokenizer.from_pretrained(\"FuseAI/FuseChat-7B-VaRM\")\n# Single-turn\ntokens = tokenizer(\"GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant:\").input_ids\nassert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]\n# Multi-turn\ntokens = tokenizer(\"GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant: Hi<|end_of_turn|>GPT4 Correct User: How are you today?<|end_of_turn|>GPT4 Correct Assistant:\").input_ids\nassert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]\n```\n\nThe GPT4 template is also available as the integrated `tokenizer.chat_template`, which can be used instead of manually specifying the template:\n\n```python\nmessages = [\n    {\"role\": \"user\", \"content\": \"Hello\"},\n    {\"role\": \"assistant\", \"content\": \"Hi\"},\n    {\"role\": \"user\", \"content\": \"How are you today?\"}\n]\ntokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)\nassert tokens == [1, 420, 6316, 28781, 3198, 3123, 1247, 28747, 22557, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747, 15359, 32000, 420, 6316, 28781, 3198, 3123, 1247, 28747, 1602, 460, 368, 3154, 28804, 32000, 420, 6316, 28781, 3198, 3123, 21631, 28747]\n```\n\n## Data Construction\n\nWe curated a comprehensive training dataset, [FuseChat-Mixture](https://huggingface.co/datasets/FuseAI/FuseChat-Mixture), from various sources. This dataset covers different styles and capabilities, featuring both human-written and model-generated, and spanning general instruction-following and specific skills. \n\nHere we show the scripts to obtain representations from multiple source LLMs for model fusion.\n\n1. Get representations for each source LLM\n\n```bash\n# We split the dataset into 4 splits, then process each split on one or multiple GPU.\n\n# OpenChat-3.5-7B\nexport CUDA_VISIBLE_DEVICES=0\nfor i in {0..3}; do\npython /train/get_data_representation.py \\\n  --model_name_or_path \"openchat/openchat_3.5\" \\\n  --data_path \"/data/fusechat_v1_clean_split_2048_filter_wrong.json\" \\\n  --dataset_save_dir \"<${i}_4_path_to_openchat_representation>\" \\\n  --tknz_dataset_path \"<${i}_4_path_to_openchat_tknz>\" \\\n  --cache_dir \"/.cache/huggingface/datasets\" \\\n  --model_max_length 2048 \\\n  --load_in_half bf16 \\\n  --batch_size 32 \\\n  --top_k_logits 10 \\\n  --save_per_token_metric \\\n  --no_assert \\\n  --conv_temp \"openchat\" \\\n  --flash_attn_transformers \\\n  --mask_instruction \\\n  --dataset_split_num 4 \\\n  --dataset_index ${i}\ndone \n\n# NH2-Mixtral-8x7B\nexport CUDA_VISIBLE_DEVICES=0,1,2\nfor i in {0..3}; do\npython /train/get_data_representation.py \\\n  --model_name_or_path \"NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO\" \\\n  --data_path \"/data/fusechat_v1_clean_split_2048_filter_wrong.json\" \\\n  --dataset_save_dir \"<${i}_4_path_to_mixtral_representation>\" \\\n  --tknz_dataset_path \"<${i}_4_path_to_mixtral_tknz>\" \\\n  --cache_dir \"/.cache/huggingface/datasets\" \\\n  --model_max_length 2048 \\\n  --load_in_half bf16 \\\n  --batch_size 4 \\\n  --top_k_logits 10 \\\n  --save_per_token_metric \\\n  --no_assert \\\n  --conv_temp \"openchat\" \\\n  --flash_attn_transformers \\\n  --mask_instruction \\\n  --device_map \"auto\" \\\n  --dataset_split_num 4 \\\n  --dataset_index ${i}\ndone \n\n# NH2-Solar-10.7B\nexport CUDA_VISIBLE_DEVICES=0\nfor i in {0..3}; do\npython /train/get_data_representation.py \\\n  --model_name_or_path \"NousResearch/Nous-Hermes-2-SOLAR-10.7B\" \\\n  --data_path \"/data/fusechat_v1_clean_split_2048_filter_wrong.json\" \\\n  --dataset_save_dir \"<${i}_4_path_to_solar_representation>\" \\\n  --tknz_dataset_path \"<${i}_4_path_to_solar_tknz>\" \\\n  --cache_dir \"/.cache/huggingface/datasets\" \\\n  --model_max_length 2048 \\\n  --load_in_half bf16 \\\n  --batch_size 8 \\\n  --top_k_logits 10 \\\n  --save_per_token_metric \\\n  --no_assert \\\n  --conv_temp \"openchat\" \\\n  --flash_attn_transformers \\\n  --mask_instruction \\\n  --dataset_split_num 4 \\\n  --dataset_index ${i}\ndone \n```\n\n2. Align representations from different source LLMs\n\n```bash\n# Since the tokenizers and vocabularies of these source LLMs are identical, we do not align.\n\n# OpenChat-3.5-7B <-> NH2-Mixtral-8x7B\nfor i in {0..3}; do\npython /train/replace_model.py \\\n  --dataset_dir \"<${i}_4_path_to_openchat_representation>\" \\\n  --replace_dataset_dir \"<${i}_4_path_to_mixtral_representation>\" \\\n  --dataset_save_dir \"<${i}_4_path_to_openchat_mixtral_representation>\" \\\n  --preprocessing_num_workers 64 \\\n  --batch_size 1000 \\\n  --replace_model model_0\ndone \n\n# OpenChat-3.5-7B <-> NH2-Solar-10.7B\nfor i in {0..3}; do\npython /train/replace_model.py \\\n  --dataset_dir \"<${i}_4_path_to_openchat_mixtral_representation>\" \\\n  --replace_dataset_dir \"<${i}_4_path_to_solar_representation>\" \\\n  --dataset_save_dir \"<${i}_4_path_to_openchat_mixtral_solar_representation>\" \\\n  --preprocessing_num_workers 64 \\\n  --batch_size 1000 \\\n  --replace_model model_1\ndone\n```\n\n3. Filter instances with NaN loss in the dataset\n\n```bash\nfor i in {0..3}; do\npython /train/filter_nan.py \\\n  --input_data_dir \"<${i}_4_path_to_openchat_mixtral_solar_representation>\" \\\n  --output_data_dir \"<${i}_4_path_to_openchat_mixtral_solar_representation_fnan>\"\ndone\n```\n\nThe final processed data is at `<${i}_4_path_to_openchat_mixtral_solar_representation_fnan>`.\n\n## Pairwise Knowledge Fusion\n\nWe show the scripts for pairwise knowledge fusion.\n\n```bash\n# OpenChat-3.5-7B <-> NH2-Mixtral-8x7B\nexport CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\ntorchrun --nproc_per_node=8 --master_port=20001 /train/train.py \\\n  --model_name_or_path \"openchat/openchat_3.5\" \\\n  --data_path \"<0_4_path_to_openchat_mixtral_solar_representation_fnan>,<1_4_path_to_openchat_mixtral_solar_representation_fnan>,<2_4_path_to_openchat_mixtral_solar_representation_fnan>,<3_4_path_to_openchat_mixtral_solar_representation_fnan>\" \\\n  --bf16 True \\\n  --output_dir \"<path_to_save_openchat_mixtral_ckpt>\" \\\n  --num_train_epochs 3 \\\n  --per_device_train_batch_size 4 \\\n  --per_device_eval_batch_size 4 \\\n  --gradient_accumulation_steps 4 \\\n  --evaluation_strategy \"no\" \\\n  --save_strategy \"epoch\" \\\n  --save_steps 10000 \\\n  --save_total_limit 5 \\\n  --learning_rate 5e-6 \\\n  --weight_decay 0. \\\n  --warmup_ratio 0.03 \\\n  --lr_scheduler_type \"cosine\" \\\n  --logging_steps 1 \\\n  --fsdp \"full_shard auto_wrap\" \\\n  --fsdp_transformer_layer_cls_to_wrap 'MistralDecoderLayer' \\\n  --tf32 True \\\n  --model_max_length 2048 \\\n  --gradient_checkpointing True \\\n  --conv_temp \"openchat\" \\\n  --lazy_preprocess True \\\n  --flash_attn_transformers True \\\n  --do_train \\\n  --do_distill \\\n  --distill_with_ref_model True \\\n  --distill_with_aligned_model_0 True \\\n  --distill_with_aligned_model_1 False \\\n  --distill_loss_type \"ce\" \\\n  --distill_teacher_temperature 1.0 \\\n  --lm_loss_weight 0.9 \\\n  --distill_greater_as_gt True \\\n  --distill_greater_as_gt_type hard \\\n  --dataloader_num_workers 8 \\\n  --remove_unused_columns False\n\n# OpenChat-3.5-7B <-> NH2-Solar-10.7B\nexport CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\ntorchrun --nproc_per_node=8 --master_port=20001 /train/train.py \\\n  --model_name_or_path \"openchat/openchat_3.5\" \\\n  --data_path \"<0_4_path_to_openchat_mixtral_solar_representation_fnan>,<1_4_path_to_openchat_mixtral_solar_representation_fnan>,<2_4_path_to_openchat_mixtral_solar_representation_fnan>,<3_4_path_to_openchat_mixtral_solar_representation_fnan>\" \\\n  --bf16 True \\\n  --output_dir \"<path_to_save_openchat_solar_ckpt>\" \\\n  --num_train_epochs 3 \\\n  --per_device_train_batch_size 4 \\\n  --per_device_eval_batch_size 4 \\\n  --gradient_accumulation_steps 4 \\\n  --evaluation_strategy \"no\" \\\n  --save_strategy \"epoch\" \\\n  --save_steps 10000 \\\n  --save_total_limit 5 \\\n  --learning_rate 5e-6 \\\n  --weight_decay 0. \\\n  --warmup_ratio 0.03 \\\n  --lr_scheduler_type \"cosine\" \\\n  --logging_steps 1 \\\n  --fsdp \"full_shard auto_wrap\" \\\n  --fsdp_transformer_layer_cls_to_wrap 'MistralDecoderLayer' \\\n  --tf32 True \\\n  --model_max_length 2048 \\\n  --gradient_checkpointing True \\\n  --conv_temp \"openchat\" \\\n  --lazy_preprocess True \\\n  --flash_attn_transformers True \\\n  --do_train \\\n  --do_distill \\\n  --distill_with_ref_model True \\\n  --distill_with_aligned_model_0 False \\\n  --distill_with_aligned_model_1 True \\\n  --distill_loss_type \"ce\" \\\n  --distill_teacher_temperature 1.0 \\\n  --lm_loss_weight 0.9 \\\n  --distill_greater_as_gt True \\\n  --distill_greater_as_gt_type hard \\\n  --dataloader_num_workers 8 \\\n  --remove_unused_columns False\n```\n\n## Model Merging\n\nWe show the scripts to obtain the final FuseChat using different merging methods.\n\n```bash\n# For \"slerp\", \"ta\", \"ties\", and \"dare\" methods (Please install \"mergekit\")\nexport CUDA_VISIBLE_DEVICES=0\nmergekit-yaml merge/mergekit_configs/fusechat-slerp.yml \"<path_to_save_fusechat_7b_slerp>\"\nmergekit-yaml merge/mergekit_configs/fusechat-ta.yml \"<path_to_save_fusechat_7b_ta>\"\nmergekit-yaml merge/mergekit_configs/fusechat-ties.yml \"<path_to_save_fusechat_7b_ties>\"\nmergekit-yaml merge/mergekit_configs/fusechat-dare.yml \"<path_to_save_fusechat_7b_dare>\"\n\n# For \"linear\" method \npython merge/VaRM/merge.py \\\n  --merged_model_names \"FuseAI/OpenChat-3.5-7B-Mixtral,FuseAI/OpenChat-3.5-7B-Solar\" \\\n  --merged_model_save_dir \"<path_to_save_fusechat_7b_linear>\" \\\n  --merge_method \"linear\" \\\n  --linear_weights \"1,2\"\n\n# For our \"varm\" method\npython merge/VaRM/analysis.py \\\n  --model1_path \"FuseAI/OpenChat-3.5-7B-Mixtral\" \\\n  --model2_path \"FuseAI/OpenChat-3.5-7B-Solar\" \\\n  --save_path \"<path_to_save_analysis_result>/analysis.json\" \\\n  --merge_type \"square\"\n\npython merge/VaRM/merge.py \\\n  --merged_model_names \"FuseAI/OpenChat-3.5-7B-Mixtral,FuseAI/OpenChat-3.5-7B-Solar\" \\\n  --analysis_result \"<path_to_save_analysis_result>/analysis.json\" \\\n  --merged_model_save_dir \"<path_to_save_fusechat_7b_varm>\" \\\n  --merge_method \"avg_param\" \\\n  --merge_type \"square\"\n```\n\n## Evaluation\n\nWe evaluate FuseChat on MT-Bench, which comprises 80 multi-turn dialogues spanning writing, roleplay, reasoning, math, coding, stem, and humanities domains. Please download the [official code](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) and follow the guidelines for evaluation. We provide the scripts for our evaluation. \n\n```bash\n# Step 1. Generate model answers to MT-bench questions\nexport CUDA_VISIBLE_DEVICES=0,1\npython gen_model_answer.py \\\n  --model-path \"FuseAI/FuseChat-7B-VaRM\" \\\n  --model-id \"openchat_3.5_fusechat_7b_varm\" \\\n  --num-gpus-per-model 1 \\\n  --num-gpus-total 2\n\n# Step 2. Generate GPT-4 judgments\nexport OPENAI_API_KEY=XXXXXX  # set the OpenAI API key\npython gen_judgment.py \\\n  --parallel 2\n\n# Step 3. Show MT-bench scores\npython show_result.py\n```\n\n## Citation\n\nIf you find this work is relevant with your research or applications, please feel free to cite our work!\n```\n@article{wan2024fusechat,\n  title={FuseChat: Knowledge Fusion of Chat Models},\n  author={Fanqi Wan and Ziyi Yang and Longguang Zhong and Xiaojun Quan and Xinting Huang and Wei Bi},\n  journal={arXiv preprint arXiv:2402.16107},\n  year={2024}\n}\n```\n\n# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)\nDetailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_FuseAI__FuseChat-7B-VaRM)\n\n|             Metric              |Value|\n|---------------------------------|----:|\n|Avg.                             |66.52|\n|AI2 Reasoning Challenge (25-Shot)|62.88|\n|HellaSwag (10-Shot)              |84.25|\n|MMLU (5-Shot)                    |63.71|\n|TruthfulQA (0-shot)              |45.67|\n|Winogrande (5-shot)              |79.16|\n|GSM8k (5-shot)                   |63.46|\n\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2402.16107",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 533,
  "gated": false,
  "private": false,
  "last_modified": "2024-09-09T12:16:03.000Z",
  "created_at": "2024-09-09T06:52:35.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66de9b3325486dfa8fdeb31c",
  "id": "RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf",
  "modelId": "RichardErkhov/FuseAI_-_FuseChat-7B-VaRM-gguf",
  "sha": "e8d15534842d9fd9128b47698cd255457842f909",
  "createdAt": "2024-09-09T06:52:35.000Z",
  "lastModified": "2024-09-09T12:16:03.000Z",
  "author": "RichardErkhov",
  "downloads": 533,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}