richarderkhov/rinna_-_llama-3-youko-8b-instruct-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "./rinna.png",
    "summary": "!rinna-icon # Overview The model is the instruction-tuned version of rinna/llama-3-youko-8b, using supervised fine-tuning (SFT), Chat Vector, and direct preference optimization (DPO). It adpots the Llama-3 chat format. | Size | Continual Pre-Training | Instruction-Tuning | | :-   | :-                     | :-                 | | 8B   | Llama 3 Youko 8B [[HF]](https://huggingface.co/rinna/llama-3-youko-8b) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-8b-gptq) | Llama 3 Youko 8B Instruct [[HF]](https://huggingface.co/rinna/llama-3-youko-8b-instruct) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-8b-instruct-gptq) | | 70B  | Llama 3 Youko 70B [[HF]](https://huggingface.co/rinna/llama-3-youko-70b) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-70b-gptq) | Llama 3 Youko 70B Instruct [[HF]](https://huggingface.co/rinna/llama-3-youko-70b-instruct) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-70b-instruct-gptq) | * **Model architecture** A 32-layer, 4096-hidden-size transformer-based language model. Refer to the Llama 3 Model Card for architecture details. * **Training: Built with Meta Llama 3** **Supervised fine-tuning.** The supervised fine-tuning data is a subset of the following datasets. **Model merging.** The fine-tuned model (llama-3-youko-8b-sft) has been enhanced through the following chat vector addition. The chat vector was obtained by subtracting the parameter vectors of meta-llama/Meta-Llama-3-8B from those of meta-llama/Meta-Llama-3-8B-Instruct. ~~~~text llama-3-youko-8b-sft + 0.5 * (meta-llama/Meta-Llama-3-8B-Instruct - meta-llama/Meta-Llama-3-8B) ~~~~ Here, the embedding layer was skipped while subtracting and adding the parameter vectors. **Direct preference optimization**  was then applied with a subset of the following datasets to build this instruct model. * **Contributors** --- # Benchmarking Please refer to rinna's LM benchmark page. --- # How to use the model We found this instruction-tuned model tends to generate repeated text more often than its base counterpart, and thus we set repetition_penalty=1.1 for better generation performance. The same repetition penalty was applied to the instruction-tuned model in the aforementioned evaluation experiments. ~~~~python import torch from transformers import AutoTokenizer, AutoModelForCausalLM model_id = \"rinna/llama-3-youko-8b-instruct\" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map=\"auto\", ) messages = [ {\"role\": \"system\", \"content\": \"あなたは誠実で優秀なアシスタントです。どうか、簡潔かつ正直に答えてください。\"}, {\"role\": \"user\", \"content\": \"西田幾多郎とはどんな人物ですか？\"}, ] input_ids = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors=\"pt\" ).to(model.device) terminators = [ tokenizer.convert_tokens_to_ids(\"\"), tokenizer.convert_tokens_to_ids(\"\") ] outputs = model.generate( input_ids, max_new_tokens=512, eos_token_id=terminators, do_sample=True, temperature=0.6, top_p=0.9, repetition_penalty=1.1, ) response = outputs[0][input_ids.shape[-1]:] response = tokenizer.decode(response, skip_special_tokens=True) print(response) ~~~~ --- # Tokenization The model uses the original meta-llama/Meta-Llama-3-8B-Instruct tokenizer. --- # How to cite ``bibtex @misc{rinna-llama-3-youko-8b-instruct, title = {rinna/llama-3-youko-8b-instruct}, author = {Chen, Xinqi and Mitsuda, Koh and Wakatsuki, Toshiaki and Sawada, Kei}, url = {https://huggingface.co/rinna/llama-3-youko-8b-instruct} } @inproceedings{sawada2024release, title = {Release of Pre-Trained Models for the {J}apanese Language}, author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh}, booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}, month = {5}, year = {2024}, pages = {13898--13905}, url = {https://aclanthology.org/2024.lrec-main.1213}, note = {\\url{https://arxiv.org/abs/2404.01657}} } ` --- # References `bibtex @article{llama3modelcard, title = {Llama 3 Model Card}, author = {AI@Meta}, year = {2024}, url = {https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md} } @article{huang2023chat, title = {Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages}, author = {Huang, Shih-Cheng and Li, Pin-Zu and Hsu, Yu-Chi and Chen, Kuang-Ming and Lin, Yu Tung and Hsiao, Shih-Kai and Tzong-Han Tsai, Richard and Lee, Hung-yi}, year = {2023}, url = {https://arxiv.org/abs/2310.04799} } `` --- # License Meta Llama 3 Community License",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nllama-3-youko-8b-instruct - GGUF\n- Model creator: https://huggingface.co/rinna/\n- Original model: https://huggingface.co/rinna/llama-3-youko-8b-instruct/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [llama-3-youko-8b-instruct.Q2_K.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q2_K.gguf) | Q2_K | 2.96GB |\n| [llama-3-youko-8b-instruct.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.IQ3_XS.gguf) | IQ3_XS | 3.28GB |\n| [llama-3-youko-8b-instruct.IQ3_S.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.IQ3_S.gguf) | IQ3_S | 3.43GB |\n| [llama-3-youko-8b-instruct.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q3_K_S.gguf) | Q3_K_S | 3.41GB |\n| [llama-3-youko-8b-instruct.IQ3_M.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.IQ3_M.gguf) | IQ3_M | 3.52GB |\n| [llama-3-youko-8b-instruct.Q3_K.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q3_K.gguf) | Q3_K | 3.74GB |\n| [llama-3-youko-8b-instruct.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q3_K_M.gguf) | Q3_K_M | 3.74GB |\n| [llama-3-youko-8b-instruct.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q3_K_L.gguf) | Q3_K_L | 4.03GB |\n| [llama-3-youko-8b-instruct.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.IQ4_XS.gguf) | IQ4_XS | 4.18GB |\n| [llama-3-youko-8b-instruct.Q4_0.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q4_0.gguf) | Q4_0 | 4.34GB |\n| [llama-3-youko-8b-instruct.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.IQ4_NL.gguf) | IQ4_NL | 4.38GB |\n| [llama-3-youko-8b-instruct.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q4_K_S.gguf) | Q4_K_S | 4.37GB |\n| [llama-3-youko-8b-instruct.Q4_K.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q4_K.gguf) | Q4_K | 4.58GB |\n| [llama-3-youko-8b-instruct.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q4_K_M.gguf) | Q4_K_M | 4.58GB |\n| [llama-3-youko-8b-instruct.Q4_1.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q4_1.gguf) | Q4_1 | 4.78GB |\n| [llama-3-youko-8b-instruct.Q5_0.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q5_0.gguf) | Q5_0 | 5.21GB |\n| [llama-3-youko-8b-instruct.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q5_K_S.gguf) | Q5_K_S | 5.21GB |\n| [llama-3-youko-8b-instruct.Q5_K.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q5_K.gguf) | Q5_K | 5.34GB |\n| [llama-3-youko-8b-instruct.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q5_K_M.gguf) | Q5_K_M | 5.34GB |\n| [llama-3-youko-8b-instruct.Q5_1.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q5_1.gguf) | Q5_1 | 5.65GB |\n| [llama-3-youko-8b-instruct.Q6_K.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q6_K.gguf) | Q6_K | 6.14GB |\n| [llama-3-youko-8b-instruct.Q8_0.gguf](https://huggingface.co/RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf/blob/main/llama-3-youko-8b-instruct.Q8_0.gguf) | Q8_0 | 7.95GB |\n\n\n\n\nOriginal model description:\n---\r\nthumbnail: https://github.com/rinnakk/japanese-pretrained-models/blob/master/rinna.png\r\nlicense: llama3\r\ndatasets:\r\n- CohereForAI/aya_dataset\r\n- kunishou/databricks-dolly-15k-ja\r\n- kunishou/HelpSteer-35k-ja\r\n- kunishou/HelpSteer2-20k-ja\r\n- kunishou/hh-rlhf-49k-ja\r\n- kunishou/oasst1-chat-44k-ja\r\n- kunishou/oasst2-chat-68k-ja\r\n- meta-math/MetaMathQA\r\n- OpenAssistant/oasst1\r\n- OpenAssistant/oasst2\r\n- sahil2801/CodeAlpaca-20k\r\nlanguage:\r\n- ja\r\n- en\r\ntags:\r\n- llama\r\n- llama-3\r\ninference: false\r\n---\r\n\r\n# `Llama 3 Youko 8B Instruct (rinna/llama-3-youko-8b-instruct)`\r\n\r\n![rinna-icon](./rinna.png)\r\n\r\n# Overview\r\n\r\nThe model is the instruction-tuned version of [rinna/llama-3-youko-8b](https://huggingface.co/rinna/llama-3-youko-8b), using supervised fine-tuning (SFT), Chat Vector, and direct preference optimization (DPO). It adpots the Llama-3 chat format.\r\n\r\n| Size | Continual Pre-Training | Instruction-Tuning |\r\n| :-   | :-                     | :-                 |\r\n| 8B   | Llama 3 Youko 8B [[HF]](https://huggingface.co/rinna/llama-3-youko-8b) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-8b-gptq) | Llama 3 Youko 8B Instruct [[HF]](https://huggingface.co/rinna/llama-3-youko-8b-instruct) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-8b-instruct-gptq) |\r\n| 70B  | Llama 3 Youko 70B [[HF]](https://huggingface.co/rinna/llama-3-youko-70b) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-70b-gptq) | Llama 3 Youko 70B Instruct [[HF]](https://huggingface.co/rinna/llama-3-youko-70b-instruct) [[GPTQ]](https://huggingface.co/rinna/llama-3-youko-70b-instruct-gptq) |\r\n\r\n* **Model architecture**\r\n\r\n    A 32-layer, 4096-hidden-size transformer-based language model. Refer to the [Llama 3 Model Card](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md) for architecture details.\r\n\r\n* **Training: Built with Meta Llama 3**\r\n\r\n    **Supervised fine-tuning.** The supervised fine-tuning data is a subset of the following datasets.\r\n\r\n    - [CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset)\r\n      - The JPN subset was used.\r\n    - [FLAN](https://github.com/google-research/FLAN/tree/main/flan/v2)\r\n    - [kunishou/databricks-dolly-15k-ja](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)\r\n    - [kunishou/hh-rlhf-49k-ja](https://huggingface.co/datasets/kunishou/hh-rlhf-49k-ja)\r\n    - [kunishou/oasst1-chat-44k-ja](https://huggingface.co/datasets/kunishou/oasst1-chat-44k-ja)\r\n    - [kunishou/oasst2-chat-68k-ja](https://huggingface.co/datasets/kunishou/oasst2-chat-68k-ja)\r\n    - [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)\r\n      - The following sections were used: MATH_AnsAug, MATH_Rephrased, MATH_SV, and MATH_FOBAR.\r\n      - The remaining sections, containing augmented data from commonly used evaluation corpora, were skipped for preventing any possibility of data leak.\r\n    - [OpenAssistant/oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1)\r\n      - The EN and JA subsets were used.\r\n    - [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2)\r\n      - The EN and JA subsets were used.\r\n    - [sahil2801/CodeAlpaca-20k](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k)\r\n    - rinna Dataset\r\n\r\n    **Model merging.** The fine-tuned model (llama-3-youko-8b-sft) has been enhanced through the following chat vector addition. The chat vector was obtained by subtracting the parameter vectors of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) from those of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct). \r\n\r\n    ~~~~text\r\n    llama-3-youko-8b-sft + 0.5 * (meta-llama/Meta-Llama-3-8B-Instruct - meta-llama/Meta-Llama-3-8B)\r\n    ~~~~\r\n\r\n    Here, the embedding layer was skipped while subtracting and adding the parameter vectors.\r\n    \r\n    **Direct preference optimization**  was then applied with a subset of the following datasets to build this instruct model.\r\n\r\n    - [kunishou/HelpSteer-35k-ja](https://huggingface.co/datasets/kunishou/HelpSteer-35k-ja)\r\n    - [kunishou/HelpSteer2-20k-ja](https://huggingface.co/datasets/kunishou/HelpSteer2-20k-ja)\r\n    - rinna Dataset\r\n  \r\n* **Contributors**\r\n\r\n    - [Xinqi Chen](https://huggingface.co/Keely0419)\r\n    - [Koh Mitsuda](https://huggingface.co/mitsu-koh)\r\n    - [Toshiaki Wakatsuki](https://huggingface.co/t-w)\r\n    - [Kei Sawada](https://huggingface.co/keisawada)\r\n\r\n---\r\n\r\n# Benchmarking\r\n\r\nPlease refer to [rinna's LM benchmark page](https://rinnakk.github.io/research/benchmarks/lm/index.html).\r\n\r\n---\r\n\r\n# How to use the model\r\n\r\nWe found this instruction-tuned model tends to generate repeated text more often than its base counterpart, and thus we set repetition_penalty=1.1 for better generation performance. The same repetition penalty was applied to the instruction-tuned model in the aforementioned evaluation experiments.\r\n\r\n~~~~python\r\nimport torch\r\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\r\n\r\nmodel_id = \"rinna/llama-3-youko-8b-instruct\"\r\n\r\ntokenizer = AutoTokenizer.from_pretrained(model_id)\r\nmodel = AutoModelForCausalLM.from_pretrained(\r\n\tmodel_id,\r\n    torch_dtype=torch.bfloat16,\r\n    device_map=\"auto\",\r\n)\r\n\r\nmessages = [\r\n    {\"role\": \"system\", \"content\": \"あなたは誠実で優秀なアシスタントです。どうか、簡潔かつ正直に答えてください。\"},\r\n    {\"role\": \"user\", \"content\": \"西田幾多郎とはどんな人物ですか？\"},\r\n]\r\n\r\ninput_ids = tokenizer.apply_chat_template(\r\n\tmessages,\r\n    add_generation_prompt=True,\r\n    return_tensors=\"pt\"\r\n).to(model.device)\r\n\r\nterminators = [\r\n\ttokenizer.convert_tokens_to_ids(\"<|end_of_text|>\"),\r\n    tokenizer.convert_tokens_to_ids(\"<|eot_id|>\")\r\n]\r\n\r\noutputs = model.generate(\r\n\tinput_ids,\r\n    max_new_tokens=512,\r\n    eos_token_id=terminators,\r\n    do_sample=True,\r\n    temperature=0.6,\r\n    top_p=0.9,\r\n    repetition_penalty=1.1,\r\n)\r\n    \r\nresponse = outputs[0][input_ids.shape[-1]:]\r\nresponse = tokenizer.decode(response, skip_special_tokens=True)\r\nprint(response)\r\n~~~~\r\n\r\n---\r\n\r\n# Tokenization\r\nThe model uses the original [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) tokenizer.\r\n\r\n---\r\n\r\n# How to cite\r\n```bibtex\r\n@misc{rinna-llama-3-youko-8b-instruct,\r\n    title = {rinna/llama-3-youko-8b-instruct},\r\n    author = {Chen, Xinqi and Mitsuda, Koh and Wakatsuki, Toshiaki and Sawada, Kei},\r\n    url = {https://huggingface.co/rinna/llama-3-youko-8b-instruct}\r\n}\r\n\r\n@inproceedings{sawada2024release,\r\n    title = {Release of Pre-Trained Models for the {J}apanese Language},\r\n    author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},\r\n    booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},\r\n    month = {5},\r\n    year = {2024},\r\n    pages = {13898--13905},\r\n    url = {https://aclanthology.org/2024.lrec-main.1213},\r\n    note = {\\url{https://arxiv.org/abs/2404.01657}}\r\n}\r\n```\r\n---\r\n\r\n# References\r\n```bibtex\r\n@article{llama3modelcard,\r\n    title = {Llama 3 Model Card},\r\n    author = {AI@Meta},\r\n    year = {2024},\r\n    url = {https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md}\r\n}\r\n\r\n@article{huang2023chat,\r\n    title = {Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages},\r\n    author = {Huang, Shih-Cheng and Li, Pin-Zu and Hsu, Yu-Chi and Chen, Kuang-Ming and Lin, Yu Tung and Hsiao, Shih-Kai and Tzong-Han Tsai, Richard and Lee, Hung-yi},\r\n    year = {2023},\r\n    url = {https://arxiv.org/abs/2310.04799}\r\n}\r\n```\r\n---\r\n\r\n# License\r\n[Meta Llama 3 Community License](https://llama.meta.com/llama3/license/)\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2404.01657",
    "arxiv:2310.04799",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 83,
  "gated": false,
  "private": false,
  "last_modified": "2024-08-05T08:01:33.000Z",
  "created_at": "2024-08-05T06:14:46.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66b06dd6e7c57eac7cb44078",
  "id": "RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf",
  "modelId": "RichardErkhov/rinna_-_llama-3-youko-8b-instruct-gguf",
  "sha": "8dcc09a269a18be8da373d10e2144e8eec653cd9",
  "createdAt": "2024-08-05T06:14:46.000Z",
  "lastModified": "2024-08-05T08:01:33.000Z",
  "author": "RichardErkhov",
  "downloads": 83,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}