richarderkhov/mediatek-research_-_breeze-7b-instruct-v0_1-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "MediaTek Research Breeze-7B (hereinafter referred to as Breeze-7B) is a language model family that builds on top of Mistral-7B, specifically intended for Traditional Chinese use. Breeze-7B-Base is the base model for the Breeze-7B series. It is suitable for use if you have substantial fine-tuning data to tune it for your specific use case. Breeze-7B-Instruct derives from the base model Breeze-7B-Base, making the resulting model amenable to be used as-is for commonly seen tasks. Breeze-7B-Instruct-64k is a slightly modified version of Breeze-7B-Instruct to enable a 64k-token context length. Roughly speaking, that is equivalent to 88k Traditional Chinese characters. *Update (Feb. 21st, 2024): Breeze-7B-Instruct-64k-v0_1 has been temporarily removed from Hugging Face due to its actual performance in long context tests not meeting expectations.* *Update (Mar. 7th, 2024): The current release version of Breeze-7B is v1.0. See Breeze-7B-Instruct-v1_0.* The current release version of Breeze-7B is v0.1. Practicality-wise: Performance-wise: *A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.*",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nBreeze-7B-Instruct-v0_1 - GGUF\n- Model creator: https://huggingface.co/MediaTek-Research/\n- Original model: https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Breeze-7B-Instruct-v0_1.Q2_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q2_K.gguf) | Q2_K | 2.66GB |\n| [Breeze-7B-Instruct-v0_1.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ3_XS.gguf) | IQ3_XS | 2.95GB |\n| [Breeze-7B-Instruct-v0_1.IQ3_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ3_S.gguf) | IQ3_S | 3.11GB |\n| [Breeze-7B-Instruct-v0_1.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K_S.gguf) | Q3_K_S | 3.09GB |\n| [Breeze-7B-Instruct-v0_1.IQ3_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ3_M.gguf) | IQ3_M | 3.2GB |\n| [Breeze-7B-Instruct-v0_1.Q3_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K.gguf) | Q3_K | 3.42GB |\n| [Breeze-7B-Instruct-v0_1.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K_M.gguf) | Q3_K_M | 3.42GB |\n| [Breeze-7B-Instruct-v0_1.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q3_K_L.gguf) | Q3_K_L | 3.7GB |\n| [Breeze-7B-Instruct-v0_1.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ4_XS.gguf) | IQ4_XS | 3.83GB |\n| [Breeze-7B-Instruct-v0_1.Q4_0.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_0.gguf) | Q4_0 | 3.99GB |\n| [Breeze-7B-Instruct-v0_1.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.IQ4_NL.gguf) | IQ4_NL | 4.03GB |\n| [Breeze-7B-Instruct-v0_1.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_K_S.gguf) | Q4_K_S | 4.01GB |\n| [Breeze-7B-Instruct-v0_1.Q4_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_K.gguf) | Q4_K | 4.23GB |\n| [Breeze-7B-Instruct-v0_1.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_K_M.gguf) | Q4_K_M | 4.23GB |\n| [Breeze-7B-Instruct-v0_1.Q4_1.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q4_1.gguf) | Q4_1 | 4.41GB |\n| [Breeze-7B-Instruct-v0_1.Q5_0.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_0.gguf) | Q5_0 | 4.83GB |\n| [Breeze-7B-Instruct-v0_1.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_K_S.gguf) | Q5_K_S | 4.83GB |\n| [Breeze-7B-Instruct-v0_1.Q5_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_K.gguf) | Q5_K | 4.95GB |\n| [Breeze-7B-Instruct-v0_1.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_K_M.gguf) | Q5_K_M | 4.95GB |\n| [Breeze-7B-Instruct-v0_1.Q5_1.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q5_1.gguf) | Q5_1 | 5.25GB |\n| [Breeze-7B-Instruct-v0_1.Q6_K.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q6_K.gguf) | Q6_K | 5.72GB |\n| [Breeze-7B-Instruct-v0_1.Q8_0.gguf](https://huggingface.co/RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf/blob/main/Breeze-7B-Instruct-v0_1.Q8_0.gguf) | Q8_0 | 7.41GB |\n\n\n\n\nOriginal model description:\n---\npipeline_tag: text-generation\nlicense: apache-2.0\nlanguage:\n- zh\n- en\n---\n\n# Model Card for MediaTek Research Breeze-7B-Instruct-v0_1\n\nMediaTek Research Breeze-7B (hereinafter referred to as Breeze-7B) is a language model family that builds on top of [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1), specifically intended for Traditional Chinese use.\n\n[Breeze-7B-Base](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0_1) is the base model for the Breeze-7B series. \nIt is suitable for use if you have substantial fine-tuning data to tune it for your specific use case.\n\n[Breeze-7B-Instruct](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1) derives from the base model Breeze-7B-Base, making the resulting model amenable to be used as-is for commonly seen tasks.\n\n[Breeze-7B-Instruct-64k](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0_1) is a slightly modified version of \nBreeze-7B-Instruct to enable a 64k-token context length. Roughly speaking, that is equivalent to 88k Traditional Chinese characters.\n\n*Update (Feb. 21st, 2024): Breeze-7B-Instruct-64k-v0_1 has been temporarily removed from Hugging Face due to its actual performance in long context tests not meeting expectations.*\n\n*Update (Mar. 7th, 2024): The current release version of Breeze-7B is v1.0. See [Breeze-7B-Instruct-v1_0](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v1_0).*\n\nThe current release version of Breeze-7B is v0.1.\n\nPracticality-wise:\n- Breeze-7B-Base expands the original vocabulary with additional 30,000 Traditional Chinese tokens. With the expanded vocabulary, everything else being equal, Breeze-7B operates at twice the inference speed for Traditional Chinese to Mistral-7B and Llama 7B. [See [Inference Performance](#inference-performance).]\n- Breeze-7B-Instruct can be used as is for common tasks such as Q&A, RAG, multi-round chat, and summarization.\n- In particular, Breeze-7B-Instruct-64k can perform tasks at a document level, not a chapter level.\n\n\nPerformance-wise:\n- Breeze-7B-Instruct demonstrates impressive performance in benchmarks for Traditional Chinese and English, when compared to similar sized open-source contemporaries such as Taiwan-LLM-7B/13B-chat, QWen-7B-Chat, and Yi-6B-Chat. [See [Chat Model Performance](#chat-model-performance).]\n\n\n*A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.*\n\n## Features\n\n- Breeze-7B-Base-v0_1\n  - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese\n  - 8k-token context length\n- Breeze-7B-Instruct-v0_1\n  - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese \n  - 8k-token context length\n  - Multi-turn dialogue (without special handling for harmfulness)\n- Breeze-7B-Instruct-64k-v0_1\n  - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese\n  - 64k-token context length\n  - Multi-turn dialogue (without special handling for harmfulness)\n\n## Model Details\n\n- Breeze-7B-Base-v0_1\n  - Finetuned from: [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)\n  - Model type: Causal decoder-only transformer language model\n  - Language: English and Traditional Chinese (zh-tw)\n- Breeze-7B-Instruct-v0_1\n  - Finetuned from: [MediaTek-Research/Breeze-7B-Base-v0_1](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0_1)\n  - Model type: Causal decoder-only transformer language model\n  - Language: English and Traditional Chinese (zh-tw)\n- Breeze-7B-Instruct-64k-v0_1\n  - Finetuned from: [MediaTek-Research/Breeze-7B-Instruct-v0_1](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1)\n  - Model type: Causal decoder-only transformer language model\n  - Language: English and Traditional Chinese (zh-tw)\n\n## Base Model Performance\n\n**TMMLU+**, **DRCD**, and **Table** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).\n[MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)\n and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).\n We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**. All choice problems adapt the selection by the log-likelihood.\n\n\n| Models                                       |        |↑ TMMLU+ (ACC) | DRCD (EM)   | Table (ACC) | MMLU (ACC) |\n|----------------------------------------------|--------|--------------|-------------|-------------|------------|\n|                                              |        |TC, Knowledge |TC, Reasoning|TC, Reasoning|EN, Knowledge|\n|                                              |        | 5 shot       | 3 shot      | 5 shot      | 5 shot     |\n| [Yi-34B](https://huggingface.co/01-ai/Yi-34B)| 34B    | 63.10        | 84.57       | 49.31  | 77.42      |\n| [Qwen-14B](https://huggingface.co/01-ai/Qwen/Qwen-14B)| 14B    | 51.30        | 16.95 *     | 50.69  | 68.83      |\n| [Yi-6B](https://huggingface.co/01-ai/Yi-6B) | 6B     | 49.63        | 76.61       | 34.72  | 65.35      |\n| [Qwen-7B](https://huggingface.co/01-ai/Qwen/Qwen-7B)| 7B     | 42.84        | 0.0 *       | 39.58  | 61.00      |\n| [**Breeze-7B-Base-v0_1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v0_1)       | 7B     | 40.35        | 81.13        | 28.47  | 61.63      |\n| [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)| 7B     | 36.93        | 79.27        | 27.78 | 64.89      |\n\n\n\\* Few-shot learning cannot effectively guide the model to generate the proper answer.\n\n\n## Chat Model Performance\n\n**TMMLU+**, **DRCD**, **Table**, and **MT-Bench-tw** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).\n[MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)\n and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).\n **MT-Bench** source from [lmsys/mt_bench_human_judgments](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments).\n We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**. All choice problems adapt the selection by the log-likelihood.\n We use the code revised from [fastchat llm_judge](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) (GPT4 as judge) to evaluate **MT-Bench-tw** and **MT-Bench**.\n\n\n| Models                                                                                                  |        |↑ MT-Bench-tw (Score)| TMMLU+ (ACC) | TMMLU+ (ACC) | DRCD (EM)   | Table (ACC) | MT-Bench (Score) | MMLU (ACC)  | MMLU (ACC)  | \n|---------------------------------------------------------------------------------------------------------|--------|--------------------|--------------|--------------|-------------|-------------|------------------|-------------|-------------|\n|                                                                                                         |        |TC, Chat            |TC, Knowledge |TC, Knowledge |TC, Reasoning|TC, Reasoning|EN, Chat          |EN, Knowledge|EN, Knowledge|\n|                                                                                                         |        |0 shot              | 0 shot       | 5 shot       | 3 shot      | 0 shot      |0 shot            |  0 shot     | 5 shot      | \n| [gpt-3.5-turbo](https://openai.com)                                                                     |        |7.1                 | 43.56        |              |             | 45.14       |7.9               |  67.09      |             |    \n| [Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat)                                                 | 34B    |6.9                 | 54.87        |              |             | 36.81       |7.6               |   71.04     |             |    \n| [Qwen-14B-Chat](https://huggingface.co/Qwen/Qwen-14B-Chat)                                              | 14B    |6.4                 | 48.41        |              |             | 41.67       |7.2               |    64.91    |             |    \n| [**Breeze-7B-Instruct-v0_1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v0_1)         | 7B     |5.7                 | 41.61        |              |             | 45.83       |7.1               |    63.26    |             |    \n| [**Breeze-7B-Instruct-64k-v0_1**](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-64k-v0_1) | 7B     |5.5                 | 40.99        |              |             | 36.11       |7.1               |    63.68    |             |    \n| [Qwen-7B-Chat](https://huggingface.co/Qwen/Qwen-7B-Chat)                                                | 7B     |5.4                 | 40.02        |              |             | 33.33       |6.2               |    55.94    |             |    \n| [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)                                                   | 6B     |5.0                 | 44.79        |              |             | 25.69       |6.0               |    59.45    |             |    \n| [Taiwan-LLM-13B-v2.0-chat](https://huggingface.co/yentinglin/Taiwan-LLM-13B-v2.0-chat)                  | 13B    |5.0                 | 29.47        |              |             | 23.61       |-*                |    50.50    |             |     \n| [Taiwan-LLM-7B-v2.1-chat](https://huggingface.co/yentinglin/Taiwan-LLM-7B-v2.1-chat)                    | 7B     |4.2                 | 28.08        |              |             | 31.25       | -*               |    42.72    |             |    \n\n\\* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.\n\n\n| Details on MT-Bench-tw (0 shot):<br/>Models         | STEM    |Extraction|Reasoning| Math   | Coding  | Roleplay| Writing |Humanities|↑ AVG   |\n|-----------------------------------------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| gpt-3.5-turbo                                       |  7.8    |  6.1    |   5.1   |   6.4   |  6.2    |   8.7   |   7.4   |   9.3   |   7.1   |\n| Yi-34B-Chat                                         |  9.0    |  4.8    |   5.7   |   4.0   |  4.7    |   8.5   |   8.7   |   9.8   |   6.9   |\n| Qwen-14B-Chat                                       |  7.6    |  5.7    |   4.5   |   4.2   |  5.3    |   7.5   |   7.3   |   9.1   |   6.4   |\n| **Breeze-7B-Instruct-v0_1**                         |  6.5    |  5.6    |   3.9   |   3.6   |  4.3    |   6.9   |   5.7   |   9.3   |   5.7   |\n| **Breeze-7B-Instruct-64k-v0_1**                     |  6.1    |  5.3    |   3.7   |   2.9   |  4.2    |   7.0   |   6.7   |   8.3   |   5.5   |\n| Qwen-7B-Chat                                        |  6.6    |  4.5    |   4.8   |   2.9   |  3.6    |   6.2   |   6.8   |   8.2   |   5.4   |\n| Yi-6B-Chat                                          |  7.3    |  2.7    |   3.1   |   3.3   |  2.3    |   7.2   |   5.2   |   8.8   |   5.0   |\n| Taiwan-LLM-13B-v2.0-chat                            |  6.1    |  3.4    |   4.1   |   2.3   |  3.1    |   7.4   |   6.6   |   6.8   |   5.0   |\n| Taiwan-LLM-7B-v2.1-chat                             |  5.2    |  2.6    |   2.3   |   1.2   |  3.4    |   6.6   |   5.7   |   6.8   |   4.2   |\n\n\n| Details on TMMLU+ (0 shot):<br/>Model               | STEM         | Social Science | Humanities | Other      | ↑ AVG   |\n|-----------------------------------------------------|--------------|----------------|------------|------------|---------|\n| Yi-34B-Chat                                         | 47.65        | 64.25          | 52.73      | 54.91      | 54.87   |\n| Qwen-14B-Chat                                       | 43.83        | 55.00          | 48.55      | 46.22      | 48.41   |\n| Yi-6B-Chat                                          | 37.80        | 51.74          | 45.36      | 44.25      | 44.79   |\n| gpt-3.5-turbo                                       | 41.58        | 48.52          | 40.96      | 43.18      | 43.56   |\n| **Breeze-7B-Instruct-v0_1**                         | 37.41        | 46.81          | 42.06      | 40.16      | 41.61   |\n| **Breeze-7B-Instruct-64k-v0_1**                     | 37.88        | 46.35          | 40.31      | 39.40      | 40.99   |\n| Qwen-7B-Chat                                        | 35.44        | 46.22          | 38.35      | 40.06      | 40.02   |\n| Taiwan-LLM-13B-v2.0-chat                            | 27.74        | 33.69          | 27.03      | 29.43      | 29.47   |\n| Taiwan-LLM-7B-v2.1-chat                             | 25.58        | 31.76          | 27.36      | 27.61      | 28.08   |\n\n\n\n## Inference Performance\nIn this test, we use the first 700 characters of the [web article](https://health.udn.com/health/story/5976/7699252?from=udn_ch1005_main_index) as the input and ask the model to write the same article again.\nAll inferences run on 2 RTX A6000 GPUs (using `vllm`, with a tensor-parallel size of 2).\n\n| Models                                                             | ↓ Inference Time (sec)|Estimated Max Input Length (Char)|\n|--------------------------------------------------------------------|-------------------|--------------------------|\n| Yi-6B-Chat                                                         |   10.62  |   5.2k                |\n| **Breeze-7B-Instruct-v0_1**                                        |  10.74  |    11.1k                 |\n| **Breeze-7B-Instruct-64k-v0_1**                                    | 10.74       |  88.8k            |\n| Qwen-7B-Chat                                                       |   10.86         |    9.8k                  |\n| Qwen-14B-Chat                                                      |   18.89  |    9.8k                  |\n| Mistral-7B-v0.1-Instruct                                           |  20.48   |    5.1k                 |\n| Taiwan-LLM-7B-v2.1-chat                                            |   26.26          |    2.2k                  |\n| Taiwan-LLM-13B-v2.0-chat                                           |   36.80          |    2.2k                  |\n| Yi-34B-Chat                                                        |  43.71   |    4.5k                  |\n\n## Long-context Performance\n\nTBD\n\n## Use in Transformers\n\nFirst install direct dependencies:\n```\npip install transformers torch accelerate\n```\nIf you want faster inference using flash-attention2, you need to install these dependencies:\n```bash\npip install packaging ninja\npip install flash-attn\n```\nThen load the model in transformers:\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"MediaTek-Research/Breeze-7B-Instruct-v0_1\",\n    device_map=\"auto\",\n    torch_dtype=torch.bfloat16,\n    attn_implementation=\"flash_attention_2\" # optional\n)\n```\n\nThe structure of the query is \n```txt\n<s>SYS_PROMPT   [INST] QUERY1 [/INST] RESPONSE1 [INST] QUERY2 [/INST] \n```\nwhere `SYS_PROMPT`, `QUERY1`, `RESPONSE1`, and `QUERY2` can be provided by the user.\n\nThe suggested default `SYS_PROMPT` is \n```txt\nYou are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.\n```\n\nWe also integrate `chat_template` into [tokenizer_config.json](tokenizer_config.json), so you can `apply_chat_template` to get the prompt.\n\n```python\n>>> from transformers import AutoTokenizer\n>>> tokenizer = AutoTokenizer.from_pretrained(\"MediaTek-Research/Breeze-7B-Instruct-v0_1\")\n>>> chat = [\n...   {\"role\": \"user\", \"content\": \"你好，請問你可以完成什麼任務？\"},\n...   {\"role\": \"assistant\", \"content\": \"你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。\"},\n...   {\"role\": \"user\", \"content\": \"太棒了！\"},\n... ]\n>>> tokenizer.apply_chat_template(chat, tokenize=False)\n\"<s>You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.   [INST] 你好，請問你可以完成什麼任務？ [/INST] 你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。 [INST] 太棒了！ [/INST] \"\n# Tokenized results\n# ['▁', '你好', '，', '請問', '你', '可以', '完成', '什麼', '任務', '？']\n# ['▁', '你好', '，', '我', '可以', '幫助', '您', '解決', '各種', '問題', '、', '提供', '資訊', '和', '協助', '您', '完成', '許多', '不同', '的', '任務', '。', '例如', '：', '回答', '技術', '問題', '、', '提供', '建議', '、', '翻譯', '文字', '、', '尋找', '資料', '或', '協助', '您', '安排', '行程', '等', '。', '請', '告訴', '我', '如何', '能', '幫助', '您', '。']\n# ['▁', '太', '棒', '了', '！']\n```\n\n## Citation\n\n```\n@article{MediaTek-Research2024breeze7b,\n      title={Breeze-7B Technical Report}, \n      author={Chan-Jan Hsu and Chang-Le Liu and Feng-Ting Liao and Po-Chun Hsu and Yi-Chang Chen and Da-Shan Shiu},\n      year={2024},\n      eprint={2403.02712},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2403.02712",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 127,
  "gated": false,
  "private": false,
  "last_modified": "2024-05-16T01:26:12.000Z",
  "created_at": "2024-05-15T23:57:36.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66454bf0eccd39a7db32045f",
  "id": "RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf",
  "modelId": "RichardErkhov/MediaTek-Research_-_Breeze-7B-Instruct-v0_1-gguf",
  "sha": "f6dde6f8bb5802562a0d389a750fc63e573eb50f",
  "createdAt": "2024-05-15T23:57:36.000Z",
  "lastModified": "2024-05-16T01:26:12.000Z",
  "author": "RichardErkhov",
  "downloads": 127,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}