richarderkhov/princeton-nlp_-_gemma-2-9b-it-dpo-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "This model was trained under the same setup as gemma-2-9b-it-SimPO, with the DPO objective. SimPO (Simple Preference Optimization) is an offline preference optimization algorithm designed to enhance the training of large language models (LLMs) with preference optimization datasets. SimPO aligns the reward function with the generation likelihood, eliminating the need for a reference model and incorporating a target reward margin to boost performance. Please refer to our preprint and github repo for more details.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\ngemma-2-9b-it-DPO - GGUF\n- Model creator: https://huggingface.co/princeton-nlp/\n- Original model: https://huggingface.co/princeton-nlp/gemma-2-9b-it-DPO/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [gemma-2-9b-it-DPO.Q2_K.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q2_K.gguf) | Q2_K | 3.54GB |\n| [gemma-2-9b-it-DPO.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.IQ3_XS.gguf) | IQ3_XS | 3.86GB |\n| [gemma-2-9b-it-DPO.IQ3_S.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.IQ3_S.gguf) | IQ3_S | 4.04GB |\n| [gemma-2-9b-it-DPO.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q3_K_S.gguf) | Q3_K_S | 4.04GB |\n| [gemma-2-9b-it-DPO.IQ3_M.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.IQ3_M.gguf) | IQ3_M | 4.19GB |\n| [gemma-2-9b-it-DPO.Q3_K.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q3_K.gguf) | Q3_K | 4.43GB |\n| [gemma-2-9b-it-DPO.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q3_K_M.gguf) | Q3_K_M | 4.43GB |\n| [gemma-2-9b-it-DPO.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q3_K_L.gguf) | Q3_K_L | 4.78GB |\n| [gemma-2-9b-it-DPO.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.IQ4_XS.gguf) | IQ4_XS | 4.86GB |\n| [gemma-2-9b-it-DPO.Q4_0.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q4_0.gguf) | Q4_0 | 5.07GB |\n| [gemma-2-9b-it-DPO.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.IQ4_NL.gguf) | IQ4_NL | 5.1GB |\n| [gemma-2-9b-it-DPO.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q4_K_S.gguf) | Q4_K_S | 5.1GB |\n| [gemma-2-9b-it-DPO.Q4_K.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q4_K.gguf) | Q4_K | 5.37GB |\n| [gemma-2-9b-it-DPO.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q4_K_M.gguf) | Q4_K_M | 5.37GB |\n| [gemma-2-9b-it-DPO.Q4_1.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q4_1.gguf) | Q4_1 | 5.55GB |\n| [gemma-2-9b-it-DPO.Q5_0.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q5_0.gguf) | Q5_0 | 6.04GB |\n| [gemma-2-9b-it-DPO.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q5_K_S.gguf) | Q5_K_S | 6.04GB |\n| [gemma-2-9b-it-DPO.Q5_K.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q5_K.gguf) | Q5_K | 6.19GB |\n| [gemma-2-9b-it-DPO.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q5_K_M.gguf) | Q5_K_M | 6.19GB |\n| [gemma-2-9b-it-DPO.Q5_1.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q5_1.gguf) | Q5_1 | 6.52GB |\n| [gemma-2-9b-it-DPO.Q6_K.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q6_K.gguf) | Q6_K | 7.07GB |\n| [gemma-2-9b-it-DPO.Q8_0.gguf](https://huggingface.co/RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf/blob/main/gemma-2-9b-it-DPO.Q8_0.gguf) | Q8_0 | 9.15GB |\n\n\n\n\nOriginal model description:\n---\nbase_model: google/gemma-2-9b-it\ntags:\n- alignment-handbook\n- generated_from_trainer\ndatasets:\n- princeton-nlp/gemma2-ultrafeedback-armorm\nmodel-index:\n- name: princeton-nlp/gemma-2-9b-it-DPO \n  results: []\n---\n\n# gemma-2-9b-it-DPO Model Card\n\nThis model was trained under the same setup as [gemma-2-9b-it-SimPO](https://huggingface.co/princeton-nlp/gemma-2-9b-it-SimPO), with the DPO objective.\n\nSimPO (Simple Preference Optimization) is an offline preference optimization algorithm designed to enhance the training of large language models (LLMs) with preference optimization datasets. SimPO aligns the reward function with the generation likelihood, eliminating the need for a reference model and incorporating a target reward margin to boost performance. Please refer to our [preprint](https://arxiv.org/pdf/2405.14734) and [github repo](https://github.com/princeton-nlp/SimPO) for more details.\n\n## Model Details\n\n### Model Description\n\nWe fine-tuned [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) on [princeton-nlp/gemma2-ultrafeedback-armorm](https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm) with the DPO objective.\n\n- **Developed by:** Yu Meng, Mengzhou Xia, Danqi Chen\n- **Model type:** Causal Language Model\n- **License:** gemma\n- **Finetuned from model:** [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it)\n\n### Model Sources\n\n<!-- Provide the basic links for the model. -->\n\n- **Repository:** https://github.com/princeton-nlp/SimPO\n- **Paper:** https://arxiv.org/pdf/2405.14734\n\n\n## How to Get Started with the Model\n```\nimport torch\nfrom transformers import pipeline\n\nmodel_id = \"princeton-nlp/gemma-2-9b-it-DPO\"\n\ngenerator = pipeline(\n    \"text-generation\",\n    model=model_id,\n    model_kwargs={\"torch_dtype\": torch.bfloat16},\n    device=\"cuda\",\n)\noutputs = generator([{\"role\": \"user\", \"content\": \"What's the difference between llamas and alpacas?\"}], do_sample=False, max_new_tokens=200)\nprint(outputs[0]['generated_text'])\n```\n\n## Training Details\n\n### Training Data\n\nWe use [princeton-nlp/gemma2-ultrafeedback-armorm](https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm) as the preference optimization dataset.\n\n#### Training Hyperparameters\n\nWe used the following hyperparameters:\n- learning rate: 5e-7\n- batch size: 128\n- beta: 0.01\n\nThe other hyperparameters are kept the same with our [SimPO recipe](https://github.com/princeton-nlp/SimPO/blob/main/training_configs/gemma-2-9b-it-simpo.yaml).\n\n#### Speeds, Sizes, Times\n\nFine-tuning the [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) on [princeton-nlp/gemma2-ultrafeedback-armorm](https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm) takes around 150 mins to finish on 8xH100 GPUs.\n\n## Evaluation Results\n\n\n|               models                    | AE2 LC | AE2 WR | AE2 Length |  AH  | AH Length |  GSM | GSM Length | MMLU | MMLU Length |\n|-----------------------------------|:------:|:------:|:----------:|:----:|:---------:|:----:|:----------:|:----:|:-----------:|\n|        [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it)       |  51.1  |  38.1  |    1571    | 40.8 |    545    | 87.4 |     395    | 72.7 |     515     |\n|  [princeton-nlp/gemma-2-9b-it-DPO](https://huggingface.co/princeton-nlp/gemma-2-9b-it-DPO)  |  67.8  |  65.4  |    2016    | 58.9 |    717    | 88.5 |     392    | 72.2 |     624     |\n| [princeton-nlp/gemma-2-9b-it-SimPO](https://huggingface.co/princeton-nlp/gemma-2-9b-it-SimPO) |  72.4  |  65.9  |    1833    | 59.1 |    693    | 88.0 |     341    | 72.2 |     441     |\n\n\n## Technical Specifications\n\n### Model Architecture and Objective\n\nThe model architecture is based on [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it). We use the DPO training objective.\n\n#### Hardware\n\nWe used 8xH100 GPUs for model training.\n\n#### Software\n\nTraining was done using the [alignment-handbook](https://github.com/huggingface/alignment-handbook) library.\n\n## Citation\n\ngemma model:\n```\n@article{gemma_2024,\n    title={Gemma},\n    url={https://www.kaggle.com/m/3301},\n    DOI={10.34740/KAGGLE/M/3301},\n    publisher={Kaggle},\n    author={Gemma Team},\n    year={2024}\n}\n```\n\nDPO paper:\n```\n@article{rafailov2024direct,\n  title={Direct Preference Optimization: Your language model is secretly a reward model},\n  author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D and Ermon, Stefano and Finn, Chelsea},\n  journal={Advances in Neural Information Processing Systems},\n  volume={36},\n  year={2024}\n}\n```\n\nSimPO paper:\n```\n@article{meng2024simpo,\n  title={{SimPO}: Simple preference optimization with a reference-free reward},\n  author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},\n  journal={arXiv preprint arXiv:2405.14734},\n  year={2024}\n}\n```\n\nUltraFeedback paper:\n```\n@article{cui2023ultrafeedback,\n  title={{UltraFeedback}: Boosting language models with high-quality feedback},\n  author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},\n  journal={arXiv preprint arXiv:2310.01377},\n  year={2023}\n}\n```\n\nArmoRM paper:\n```\n@article{wang2024interpretable,\n  title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},\n  author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},\n  journal={arXiv preprint arXiv:2406.12845},\n  year={2024}\n}\n```\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2405.14734",
    "arxiv:2310.01377",
    "arxiv:2406.12845",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 280,
  "gated": false,
  "private": false,
  "last_modified": "2024-07-28T10:11:46.000Z",
  "created_at": "2024-07-28T05:59:41.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66a5de4d5721968cd326a4cb",
  "id": "RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf",
  "modelId": "RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf",
  "sha": "f3f17f2602d3965c67d335628af8cd8de58f59d2",
  "createdAt": "2024-07-28T05:59:41.000Z",
  "lastModified": "2024-07-28T10:11:46.000Z",
  "author": "RichardErkhov",
  "downloads": 280,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}