richarderkhov/triaiexperiments_-_sfr-iterative-dpo-llama-3-8b-r-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "I think they had the wrong license, which is why they deleted the original repos. The correct license would be the llama 3 license --- license: cc-by-nc-nd-3.0 --- # SFR-Iterative-DPO-Llama-3-8B-R",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nSFR-Iterative-DPO-LLaMA-3-8B-R - GGUF\n- Model creator: https://huggingface.co/TriAiExperiments/\n- Original model: https://huggingface.co/TriAiExperiments/SFR-Iterative-DPO-LLaMA-3-8B-R/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q2_K.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q2_K.gguf) | Q2_K | 2.96GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.IQ3_XS.gguf) | IQ3_XS | 3.28GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.IQ3_S.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.IQ3_S.gguf) | IQ3_S | 3.43GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K_S.gguf) | Q3_K_S | 3.41GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.IQ3_M.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.IQ3_M.gguf) | IQ3_M | 3.52GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K.gguf) | Q3_K | 3.74GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K_M.gguf) | Q3_K_M | 3.74GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q3_K_L.gguf) | Q3_K_L | 4.03GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.IQ4_XS.gguf) | IQ4_XS | 4.18GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_0.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_0.gguf) | Q4_0 | 4.34GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.IQ4_NL.gguf) | IQ4_NL | 4.38GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_K_S.gguf) | Q4_K_S | 4.37GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_K.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_K.gguf) | Q4_K | 4.58GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_K_M.gguf) | Q4_K_M | 4.58GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_1.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q4_1.gguf) | Q4_1 | 4.78GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_0.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_0.gguf) | Q5_0 | 5.21GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_K_S.gguf) | Q5_K_S | 5.21GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_K.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_K.gguf) | Q5_K | 5.34GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_K_M.gguf) | Q5_K_M | 5.34GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_1.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q5_1.gguf) | Q5_1 | 5.65GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q6_K.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q6_K.gguf) | Q6_K | 6.14GB |\n| [SFR-Iterative-DPO-LLaMA-3-8B-R.Q8_0.gguf](https://huggingface.co/RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf/blob/main/SFR-Iterative-DPO-LLaMA-3-8B-R.Q8_0.gguf) | Q8_0 | 7.95GB |\n\n\n\n\nOriginal model description:\n# Original Salesforce Readme\nI think they had the wrong license, which is why they deleted the original repos.\nThe correct license would be the [llama 3 license](https://github.com/meta-llama/llama3/blob/main/LICENSE)\n\n---\nlicense: cc-by-nc-nd-3.0\n---\n# SFR-Iterative-DPO-Llama-3-8B-R\n\n## Introduction\nWe release a state-of-the-art instruct model of its class, **SFR-Iterative-DPO-LLaMA-3-8B-R**.\nOn all three widely-used instruct model benchmarks: **Alpaca-Eval-V2**, **MT-Bench**, **Chat-Arena-Hard**, our model outperforms all models of similar size (e.g., LLaMA-3-8B-it), most large open-sourced models (e.g., Mixtral-8x7B-it),\nand strong proprietary models (e.g., GPT-3.5-turbo-0613). The model is trained with open-sourced datasets without any additional human-/GPT4-labeling.\n\n## Model Releases\n- [SFT model](https://huggingface.co/Salesforce/SFR-SFT-LLaMA-3-8B-R)\n- [Reward model](https://huggingface.co/Salesforce/SFR-RM-LLaMA-3-8B-R)\n- [RLHF model](https://huggingface.co/Salesforce/SFR-Iterative-DPO-LLaMA-3-8B-R)\n\n\n## Training methods\nWe have developed a simple and efficient online RLHF recipe for LLM instruct training. Our recipe is DPO-based and thus much cheaper and simpler to train and tune compared to PPO-based approaches.\nUnlike widely-used offline DPO, the online component of our approach effectively mitigates distribution shifts during policy optimization.\nFor a detailed exposition, please refer to our accompanying technical report.\n\n\n## Chat Benchmarks\n\n| **Model**               | **Size** | **Method**        | **LC Alpaca-Eval-V2** | **MT-Bench** | **Chat-Arena-Hard** |\n|-------------------------|----------|-------------------|-----------------------|--------------|---------------------|\n| **Small Open-Sourced Models**           |          |                   |                       |              |                     |\n| Gemma-7B-it             | 7B       | SFT               | 10.4                  | 6.38         | 7.5                 |\n| Zephyr-7B-beta          | 7B       | Vanilla DPO       | 13.1                  | 7.34         | -                   |\n| Mistral-7B-v0.2-it      | 7B       | SFT               | 17.1                  | 7.51         | 12.6                |\n| Open-Chat-0106          | 7B       | SFT               | 15.6                  | 7.8          | -                   |\n| Starling-7B-beta        | 7B       | PPO               | 25.8                  | 8.12         | 23.0                |\n| LLaMA-3-8B-it           | 8B       | RS+DPO+PPO        | 22.9                  | 8.16         | 20.6                |\n| **Ours**                |          |                   |                       |              |                     |\n| Ours (SFT baseline)     | 8B       | SFT               | 10.2                  | 7.69         | 5.6                 |\n| Ours (DPO baseline)     | 8B       | Vanilla DPO       | 22.5                  | 8.17         | 22.4                |\n| Ours (Online RLHF)      | 8B       | Iterative DPO     | **37.2**              | **8.46**     | **29.1**            |\n| **Large Open-Sourced Models**       |          |                   |                       |              |                     |\n| Vicuna-33b-v1.3         | 33B      | SFT               | 17.6                  | 7.12         | 8.6                 |\n| Yi-34B-Chat             | 34B      | SFT               | 27.2                  | -            | 23.1                |\n| Mixtral-8x7B-it         | 45B*     | SFT               | 23.7                  | 8.30         | 23.4                |\n| Tulu-2-DPO-70B          | 70B      | Vanilla DPO       | 21.2                  | 7.89         | 15.0                |\n| LLaMA-3-70B-it          | 70B      | RS+DPO+PPO        | 34.4                  | 8.95         | 41.1                |\n| Mixtral-8x22B-it        | 141B*    | SFT               | 30.9                  | 8.66         | 36.4                |\n| **Proprietary Models**  |       |                   |                       |              |                     |\n| GPT-3.5-turbo-1106      | -        | -                 | 19.3                  | 8.35         | 18.9                |\n| GPT-3.5-turbo-0613      | -        | -                 | 22.7                  | 8.39         | 24.8                |\n| GPT-4-0613              | -        | -                 | 30.2                  | 9.18         | 37.9                |\n| Claude-3-Opus           | -        | -                 | 40.5                  | 9.00         | 60.4                |\n| GPT-4 Turbo (04/09)     | -        | -                 | 55.0                  | -            | 82.6                |\n\n\n## Academic Benchmarks\n\n| **Model**                  | **Size** | **Method**      | **GSM-8K** | **MMLU** | **HumanEval** | **TruthfulQA** | **ARC** | **MBPP** |\n|----------------------------|----------|-----------------|------------|----------|---------------|----------------|---------|----------|\n| LLaMA-3-8B-it              | 8B       | RS+DPO+PPO      | 79.6       | 66.0     | 61.6          | 43.9           | 59.5    | 61.1     |\n| Ours (SFT baseline)        | 8B       | SFT             | 74.2       | 64.7     | 65.2          | 53.4           | 61.4    | 62.3     |\n| Ours (DPO baseline)        | 8B       | Vanilla DPO     | 79.8       | 64.5     | 63.4          | 61.8           | 65.2    | 60.3     |\n| Ours (Iterative RLHF)      | 8B       | Iterative DPO   | 80.7       | 65.3     | 64.6          | 60.4           | 64.3    | 60.8     |\n\n\n## Usage\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ndevice = \"cuda\" \n\nmodel = AutoModelForCausalLM.from_pretrained(\"Salesforce/SFR-Iterative-DPO-LLaMA-3-8B-R\")\ntokenizer = AutoTokenizer.from_pretrained(\"Salesforce/SFR-Iterative-DPO-LLaMA-3-8B-R\")\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"I'm trying to teach myself to have nicer handwriting. Can you help?\"},\n]\n\nmodel_inputs = tokenizer.apply_chat_template(messages, return_tensors=\"pt\")\n\nmodel_inputs = model_inputs.to(device)\nmodel.to(device)\n\noutput_tokens = model.generate(model_inputs, max_new_tokens=1024, do_sample=True)\nmodel_outputs = tokenizer.batch_decode(output_tokens)\nprint(model_outputs[0])\n```\n\n\n## Limitations\nSFR-Iterative-DPO-LLaMA-3-8B-R is a research model developed as part of our RLHF initiative at Salesforce. \nWhile safety and ethical considerations are integral to our alignment process, \nthere remains the possibility that the model could generate offensive or unethical content, particularly under adversarial conditions. \nWe are committed to continuous improvement in our models to minimize such risks and encourage responsible usage.\n\n## Citation\nPlease cite our papers if you find our models are useful.\n\n```bibtex\n@misc{dong2024rlhf,\n      title={RLHF Workflow: From Reward Modeling to Online RLHF}, \n      author={Hanze Dong and Wei Xiong and Bo Pang and Haoxiang Wang and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},\n      year={2024},\n      eprint={2405.07863},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n\n@misc{xiong2024iterative,\n      title={Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint}, \n      author={Wei Xiong and Hanze Dong and Chenlu Ye and Ziqi Wang and Han Zhong and Heng Ji and Nan Jiang and Tong Zhang},\n      year={2024},\n      eprint={2312.11456},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2405.07863",
    "arxiv:2312.11456",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 88,
  "gated": false,
  "private": false,
  "last_modified": "2024-08-21T04:31:40.000Z",
  "created_at": "2024-08-21T02:28:37.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66c550d51c480e1057fbb61d",
  "id": "RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf",
  "modelId": "RichardErkhov/TriAiExperiments_-_SFR-Iterative-DPO-LLaMA-3-8B-R-gguf",
  "sha": "4a209f35a9bec1884bb885e934948ed983235a95",
  "createdAt": "2024-08-21T02:28:37.000Z",
  "lastModified": "2024-08-21T04:31:40.000Z",
  "author": "RichardErkhov",
  "downloads": 88,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}