richarderkhov/microsoft_-_rho-math-7b-v0.1-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "https://github.com/microsoft/rho/blob/main/docs/static/images/acc_vs_tokens_1b_7b.png?raw=true",
    "summary": "Quantization made by Richard Erkhov. Github Discord Request more models rho-math-7b-v0.1 - GGUF | Name | Quant method | Size | | ---- | ---- | ---- | | rho-math-7b-v0.1.Q2_K.gguf | Q2_K | 2.53GB | | rho-math-7b-v0.1.IQ3_XS.gguf | IQ3_XS | 2.81GB | | rho-math-7b-v0.1.IQ3_S.gguf | IQ3_S | 2.96GB | | rho-math-7b-v0.1.Q3_K_S.gguf | Q3_K_S | 2.95GB | | rho-math-7b-v0.1.IQ3_M.gguf | IQ3_M | 3.06GB | | rho-math-7b-v0.1.Q3_K.gguf | Q3_K | 3.28GB | | rho-math-7b-v0.1.Q3_K_M.gguf | Q3_K_M | 3.28GB | | rho-math-7b-v0.1.Q3_K_L.gguf | Q3_K_L | 3.56GB | | rho-math-7b-v0.1.IQ4_XS.gguf | IQ4_XS | 3.67GB | | rho-math-7b-v0.1.Q4_0.gguf | Q4_0 | 3.83GB | | rho-math-7b-v0.1.IQ4_NL.gguf | IQ4_NL | 3.87GB | | rho-math-7b-v0.1.Q4_K_S.gguf | Q4_K_S | 3.86GB | | rho-math-7b-v0.1.Q4_K.gguf | Q4_K | 4.07GB | | rho-math-7b-v0.1.Q4_K_M.gguf | Q4_K_M | 4.07GB | | rho-math-7b-v0.1.Q4_1.gguf | Q4_1 | 4.24GB | | rho-math-7b-v0.1.Q5_0.gguf | Q5_0 | 4.65GB | | rho-math-7b-v0.1.Q5_K_S.gguf | Q5_K_S | 4.65GB | | rho-math-7b-v0.1.Q5_K.gguf | Q5_K | 4.78GB | | rho-math-7b-v0.1.Q5_K_M.gguf | Q5_K_M | 4.78GB | | rho-math-7b-v0.1.Q5_1.gguf | Q5_1 | 5.07GB | | rho-math-7b-v0.1.Q6_K.gguf | Q6_K | 5.53GB | Original model description: --- license: mit tags: language: pipeline_tag: text-generation ---  Rho-1: Not All Tokens Are What You Need   [📜 Arxiv] • [💬 HF Paper] • [🤗 Models] • [🐱 GitHub]     Figure 1: Rho-1 is pre-trained with Selective Language Modeling (SLM). SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nrho-math-7b-v0.1 - GGUF\n- Model creator: https://huggingface.co/microsoft/\n- Original model: https://huggingface.co/microsoft/rho-math-7b-v0.1/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [rho-math-7b-v0.1.Q2_K.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q2_K.gguf) | Q2_K | 2.53GB |\n| [rho-math-7b-v0.1.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.IQ3_XS.gguf) | IQ3_XS | 2.81GB |\n| [rho-math-7b-v0.1.IQ3_S.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.IQ3_S.gguf) | IQ3_S | 2.96GB |\n| [rho-math-7b-v0.1.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q3_K_S.gguf) | Q3_K_S | 2.95GB |\n| [rho-math-7b-v0.1.IQ3_M.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.IQ3_M.gguf) | IQ3_M | 3.06GB |\n| [rho-math-7b-v0.1.Q3_K.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q3_K.gguf) | Q3_K | 3.28GB |\n| [rho-math-7b-v0.1.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q3_K_M.gguf) | Q3_K_M | 3.28GB |\n| [rho-math-7b-v0.1.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q3_K_L.gguf) | Q3_K_L | 3.56GB |\n| [rho-math-7b-v0.1.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.IQ4_XS.gguf) | IQ4_XS | 3.67GB |\n| [rho-math-7b-v0.1.Q4_0.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q4_0.gguf) | Q4_0 | 3.83GB |\n| [rho-math-7b-v0.1.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.IQ4_NL.gguf) | IQ4_NL | 3.87GB |\n| [rho-math-7b-v0.1.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q4_K_S.gguf) | Q4_K_S | 3.86GB |\n| [rho-math-7b-v0.1.Q4_K.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q4_K.gguf) | Q4_K | 4.07GB |\n| [rho-math-7b-v0.1.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q4_K_M.gguf) | Q4_K_M | 4.07GB |\n| [rho-math-7b-v0.1.Q4_1.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q4_1.gguf) | Q4_1 | 4.24GB |\n| [rho-math-7b-v0.1.Q5_0.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q5_0.gguf) | Q5_0 | 4.65GB |\n| [rho-math-7b-v0.1.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q5_K_S.gguf) | Q5_K_S | 4.65GB |\n| [rho-math-7b-v0.1.Q5_K.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q5_K.gguf) | Q5_K | 4.78GB |\n| [rho-math-7b-v0.1.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q5_K_M.gguf) | Q5_K_M | 4.78GB |\n| [rho-math-7b-v0.1.Q5_1.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q5_1.gguf) | Q5_1 | 5.07GB |\n| [rho-math-7b-v0.1.Q6_K.gguf](https://huggingface.co/RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf/blob/main/rho-math-7b-v0.1.Q6_K.gguf) | Q6_K | 5.53GB |\n\n\n\n\nOriginal model description:\n---\nlicense: mit\ntags:\n- nlp\n- math\nlanguage:\n- en\npipeline_tag: text-generation\n---\n\n\n<h1 align=\"center\">\nRho-1: Not All Tokens Are What You Need\n</h1>\n\n\n<p align=\"center\">\n  <a href=\"https://arxiv.org/abs/2404.07965\"><b>[📜 Arxiv]</b></a> •\n  <a href=\"https://huggingface.co/papers/2404.07965\"><b>[💬 HF Paper]</b></a> •\n  <a href=\"https://huggingface.co/microsoft/rho-math-1b-v0.1\"><b>[🤗 Models]</b></a> •\n  <a href=\"https://github.com/microsoft/rho\"><b>[🐱 GitHub]</b></a>\n</p>\n\n<p align=\"center\">\n    <img src=\"https://github.com/microsoft/rho/blob/main/docs/static/images/acc_vs_tokens_1b_7b.png?raw=true\" width=\"1000\">\n        <br>\n    <em>Figure 1: Rho-1 is pre-trained with Selective Language Modeling (SLM). SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster.</em>\n</p>\n\n\n## 🔥 News\n\n- [2024/04/12] 🔥🔥🔥 Rho-Math-v0.1 models released at 🤗 HuggingFace! \n    - [Rho-Math-1B](https://huggingface.co/microsoft/rho-math-1b-v0.1) and [Rho-Math-7B](https://huggingface.co/microsoft/rho-math-7b-v0.1) achieve 15.6% and 31.0% few-shot accuracy on MATH dataset, respectively — matching DeepSeekMath with only 3\\% of the pretraining tokens.\n    - [Rho-Math-1B-Interpreter](https://huggingface.co/microsoft/rho-math-1b-interpreter-v0.1) is the first 1B LLM that achieves over 40% accuracy on MATH.\n    - [Rho-Math-7B-Interpreter](https://huggingface.co/microsoft/rho-math-7b-interpreter-v0.1) achieves 52% on MATH dataset, using only 69k samples for fine-tuning.\n- [2024/04/11] Rho-1 paper and repo released.\n\n\n\n## 💡 Introduction\n\nRho-1 base models employ Selective Language Modeling (SLM) for pretraining, which selectively trains on clean and useful tokens that aligned with the desired distribution.\n\n\n### Selective Lanugage Modeling (SLM)\n\n<p align=\"center\">\n    <img src=\"https://github.com/microsoft/rho/blob/main/docs/static/images/example.png?raw=true\" width=\"1000\">\n        <br>\n    <em>Figure 2:\n    <b>Upper:</b> Even an extensively filtered pretraining corpus contains token-level noise.\n    <b>Left:</b> Previous Causal Language Modeling (CLM) trains on all tokens.\n    <b>Right:</b> Our proposed Selective Language Modeling (SLM) selectively applies loss on those useful and clean tokens.</em>\n</p>\n\n<p align=\"center\">\n    <img src=\"https://github.com/microsoft/rho/blob/main/docs/static/images/pipeline.png?raw=true\" width=\"1000\">\n        <br>\n    <em>Figure 3: <b>The pipeline of Selective Language Modeling.</b>\n    SLM optimizes language model performance by concentrating on valuable, clean tokens during pre-training.\n    It involves three steps:\n    (Step 1) Initially, train a reference model on high-quality data.\n    (Step 2) Then, score each token's loss in a corpus using the reference model.\n    (Step 3) Finally, train the language model selectively on tokens that show higher excess loss compared to the reference loss.</em>\n</p>\n\n<!-- results: -->\n\n### Evaluation Results\n\nBase models (Few-shot CoT):\n\n|     **Model**     | **Size** | **Data** | **Uniq. Token** | **Train Token** | **GSM8K** | **MATH** | **MMLU STEM** |  **SAT** |\n|:-----------------:|:--------:|:--------:|:---------------:|:---------------:|:---------:|:--------:|:-------------:|:--------:|\n| 1-2B Base Models  |          |          |                 |                 |           |          |               |          |\n| Qwen1.5           | 1.8B     | -        | -               | -               | 36.1      | 6.8      | 31.3          | 40.6     |\n| Gemma             | 2.0B     | -        | -               | -               | 18.8      | 11.4     | **34.4**      | 50.0     |\n| DeepSeekMath      | 1.3B     | -        | 120B            | 150B            | 23.8      | 13.6     | 33.1          | **56.3** |\n| [Rho-Math-1B-v0.1](https://huggingface.co/microsoft/rho-math-1b-v0.1)  | 1.1B     | OWM      | 14B             | 30B             | **36.2**  | **15.6** | 23.3          | 28.1     |\n| >= 7B Base Models |          |          |                 |                 |           |          |               |          |\n| Mistral           | 7B       |          | -               | -               | 41.2      | 11.6     | 49.5          | 59.4     |\n| Minerva           | 540B     | -        | 39B             | 26B             | 58.8      | 33.6     | **63.9**      | -        |\n| LLemma            | 34B      | PPile    | 55B             | 50B             | 54.2      | 23.0     | 54.7          | 68.8     |\n| InternLM2-Math    | 20B      | -        | 31B             | 125B            | 65.4      | 30.0     | 53.1          | 71.9     |\n| DeepSeekMath      | 7B       | -        | 120B            | 500B            | 64.1      | **34.2** | 56.4          | **84.4** |\n| [Rho-Math-7B-v0.1](https://huggingface.co/microsoft/rho-math-7b-v0.1)  | 7B       | OWM      | 14B             | 10.5B           | **66.9**  | 31.0     | 54.6          | **84.4** |\n\n\n[Tool-integrated reasoning](https://github.com/microsoft/ToRA) (Code Interpreter):\n\n| **Model**                    | **Size** | **SFT Data** | **GSM8k** | **MATH** | **SVAMP** | **ASDiv** | **MAWPS** | **TabMWP** | **GSM-Hard** | **AVG**  |\n|------------------------------|----------|--------------|-----------|----------|-----------|-----------|-----------|------------|--------------|----------|\n| gpt4-early (pal)             | -        | -            | 94.2      | 51.8     | 94.8      | 92.6      | 97.7      | 95.9       | 77.6         | 86.4     |\n| gpt-4-turbo-2024-04-09 (cot) | - | - | - | 73.4 | - | - | - | - | - |\n| Open-Source Small Models | | | | | | | | | |\n| MAmmoTH                      | 70B      | MI-260k      | 76.9      | 41.8     | 82.4      | -         | -         | -          | -            | -        |\n| ToRA                         | 7B       | ToRA-69k     | 68.8      | 40.1     | 68.2      | 73.9      | 88.8      | 42.4       | 54.6         | 62.4     |\n| ToRA                         | 70B      | ToRA-69k     | 84.3      | 49.7     | **82.7**  | 86.8      | 93.8      | 74.0       | **67.2**     | **76.9** |\n| DeepSeekMath                 | 7B       | ToRA-69k     | 79.8      | **52.0** | 80.1      | **87.1**  | 93.8      | **85.8**   | 63.1         | 77.4     |\n| [Rho-Math-1B-Interpreter-v0.1](https://huggingface.co/microsoft/rho-math-1b-interpreter-v0.1) | 1B       | ToRA-69k     | 59.4      | 40.6     | 60.7      | 74.2      | 88.6      | 26.7       | 48.1         | 56.9     |\n| [Rho-Math-7B-Interpreter-v0.1](https://huggingface.co/microsoft/rho-math-7b-interpreter-v0.1) | 7B       | ToRA-69k     | 81.3      | **51.8** | 80.8      | 85.5      | **94.5**  | 70.1       | 63.1         | 75.3     |\n\n\n## 🚀 Quick Start\n\n\n### Evaluation\n\n```sh\ngit clone git@github.com:microsoft/rho.git\ncd rho-1/math-evaluation-harness\n```\n\nBase model few-shot evaluation:\n\n```sh\nbash scripts/run_eval.sh cot microsoft/rho-math-7b-v0.1\n```\n\nSFT model (code-interpreter) evaluation:\n\n```sh\nbash scripts/run_eval.sh tora microsoft/rho-math-7b-interpreter-v0.1\n```\n\nOur reproduced outputs are provided in `rho-1/outputs.zip`.\n\n\n\n## ☕️ Citation\n\nIf you find this repository helpful, please consider citing our paper:\n\n```\n@misc{lin2024rho1,\n      title={Rho-1: Not All Tokens Are What You Need}, \n      author={Zhenghao Lin and Zhibin Gou and Yeyun Gong and Xiao Liu and Yelong Shen and Ruochen Xu and Chen Lin and Yujiu Yang and Jian Jiao and Nan Duan and Weizhu Chen},\n      year={2024},\n      eprint={2404.07965},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2404.07965",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 859,
  "gated": false,
  "private": false,
  "last_modified": "2024-04-20T20:22:28.000Z",
  "created_at": "2024-04-20T18:12:48.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "662405a0251ee52d520ee7dd",
  "id": "RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf",
  "modelId": "RichardErkhov/microsoft_-_rho-math-7b-v0.1-gguf",
  "sha": "f07d1df42878ed2751444f62aa763199a7310bd7",
  "createdAt": "2024-04-20T18:12:48.000Z",
  "lastModified": "2024-04-20T20:22:28.000Z",
  "author": "RichardErkhov",
  "downloads": 859,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 23
}