richarderkhov/mlabonne_-_neuralhermes-2.5-mistral-7b-laser-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "https://i.imgur.com/gUlEJuU.jpeg",
    "summary": "This is an experimental LASER version of NeuralHermes using laserRMT, based on this paper. |                                                Model                                                 |AGIEval|GPT4All|TruthfulQA|Bigbench|Average| |------------------------------------------------------------------------------------------------------|------:|------:|---------:|-------:|------:| |NeuralHermes-2.5-Mistral-7B-laser|  43.54|  73.44|     55.26|   42.24|  53.62| |NeuralHermes-2.5-Mistral-7B            |  43.67|  73.24|     55.37|   41.76|  53.51| Fernando Fernandes Neto and Eric Hartford. \"Optimizing Large Language Models Using Layer-Selective Rank Reduction and Random Matrix Theory.\" 2024. NeuralHermes is an teknium/OpenHermes-2.5-Mistral-7B model that has been further fine-tuned with Direct Preference Optimization (DPO) using the mlabonne/chatml_dpo_pairs dataset. It surpasses the original model on several benchmarks (see results). It is directly inspired by the RLHF process described by Intel/neural-chat-7b-v3-1's authors to improve performance. I used the same dataset and reformatted it to apply the ChatML template. The code to train this model is available on Google Colab and GitHub. It required an A100 GPU for about an hour.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nNeuralHermes-2.5-Mistral-7B-laser - GGUF\n- Model creator: https://huggingface.co/mlabonne/\n- Original model: https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B-laser/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q2_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q2_K.gguf) | Q2_K | 2.53GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ3_XS.gguf) | IQ3_XS | 2.81GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.IQ3_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ3_S.gguf) | IQ3_S | 2.96GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K_S.gguf) | Q3_K_S | 2.95GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.IQ3_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ3_M.gguf) | IQ3_M | 3.06GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K.gguf) | Q3_K | 3.28GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K_M.gguf) | Q3_K_M | 3.28GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K_L.gguf) | Q3_K_L | 3.56GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ4_XS.gguf) | IQ4_XS | 3.67GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q4_0.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_0.gguf) | Q4_0 | 3.83GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ4_NL.gguf) | IQ4_NL | 3.87GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_K_S.gguf) | Q4_K_S | 3.86GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q4_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_K.gguf) | Q4_K | 4.07GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_K_M.gguf) | Q4_K_M | 4.07GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q4_1.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_1.gguf) | Q4_1 | 4.24GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q5_0.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_0.gguf) | Q5_0 | 4.65GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_K_S.gguf) | Q5_K_S | 4.65GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q5_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_K.gguf) | Q5_K | 4.78GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_K_M.gguf) | Q5_K_M | 4.78GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q5_1.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_1.gguf) | Q5_1 | 5.07GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q6_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q6_K.gguf) | Q6_K | 5.53GB |\n| [NeuralHermes-2.5-Mistral-7B-laser.Q8_0.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q8_0.gguf) | Q8_0 | 7.17GB |\n\n\n\n\nOriginal model description:\n---\nlanguage:\n- en\nlicense: apache-2.0\ntags:\n- mistral\n- instruct\n- finetune\n- chatml\n- gpt4\n- synthetic data\n- distillation\n- dpo\n- rlhf\n- laser\ndatasets:\n- mlabonne/chatml_dpo_pairs\nbase_model: teknium/OpenHermes-2.5-Mistral-7B\nmodel-index:\n- name: NeuralHermes-2.5-Mistral-7B-laser\n  results:\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: AI2 Reasoning Challenge (25-Shot)\n      type: ai2_arc\n      config: ARC-Challenge\n      split: test\n      args:\n        num_few_shot: 25\n    metrics:\n    - type: acc_norm\n      value: 66.38\n      name: normalized accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: HellaSwag (10-Shot)\n      type: hellaswag\n      split: validation\n      args:\n        num_few_shot: 10\n    metrics:\n    - type: acc_norm\n      value: 85.09\n      name: normalized accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: MMLU (5-Shot)\n      type: cais/mmlu\n      config: all\n      split: test\n      args:\n        num_few_shot: 5\n    metrics:\n    - type: acc\n      value: 63.43\n      name: accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: TruthfulQA (0-shot)\n      type: truthful_qa\n      config: multiple_choice\n      split: validation\n      args:\n        num_few_shot: 0\n    metrics:\n    - type: mc2\n      value: 54.95\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: Winogrande (5-shot)\n      type: winogrande\n      config: winogrande_xl\n      split: validation\n      args:\n        num_few_shot: 5\n    metrics:\n    - type: acc\n      value: 78.14\n      name: accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser\n      name: Open LLM Leaderboard\n  - task:\n      type: text-generation\n      name: Text Generation\n    dataset:\n      name: GSM8k (5-shot)\n      type: gsm8k\n      config: main\n      split: test\n      args:\n        num_few_shot: 5\n    metrics:\n    - type: acc\n      value: 55.72\n      name: accuracy\n    source:\n      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser\n      name: Open LLM Leaderboard\n---\n\n<center><img src=\"https://i.imgur.com/gUlEJuU.jpeg\"></center>\n\n# NeuralHermes 2.5 - Mistral 7B - LASER\n\nThis is an experimental LASER version of NeuralHermes using [laserRMT](https://github.com/cognitivecomputations/laserRMT), based on [this paper](https://arxiv.org/pdf/2312.13558.pdf).\n\n|                                                Model                                                 |AGIEval|GPT4All|TruthfulQA|Bigbench|Average|\n|------------------------------------------------------------------------------------------------------|------:|------:|---------:|-------:|------:|\n|[NeuralHermes-2.5-Mistral-7B-laser](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B-laser)|  43.54|  73.44|     55.26|   42.24|  53.62|\n|[NeuralHermes-2.5-Mistral-7B](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B)            |  43.67|  73.24|     55.37|   41.76|  53.51|\n\nFernando Fernandes Neto and Eric Hartford. \"Optimizing Large Language Models Using Layer-Selective Rank Reduction and Random Matrix Theory.\" 2024.\n\nNeuralHermes is an [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) model that has been further fine-tuned with Direct Preference Optimization (DPO) using the [mlabonne/chatml_dpo_pairs](https://huggingface.co/datasets/mlabonne/chatml_dpo_pairs) dataset. It surpasses the original model on several benchmarks (see results).\n\nIt is directly inspired by the RLHF process described by [Intel/neural-chat-7b-v3-1](https://huggingface.co/Intel/neural-chat-7b-v3-1)'s authors to improve performance. I used the same dataset and reformatted it to apply the ChatML template.\n\nThe code to train this model is available on [Google Colab](https://colab.research.google.com/drive/15iFBr1xWgztXvhrj5I9fBv20c7CFOPBE?usp=sharing) and [GitHub](https://github.com/mlabonne/llm-course/tree/main). It required an A100 GPU for about an hour.\n\n## Results\n\n### AGIEval\n|             Task             |Version| Metric |Value|   |Stderr|\n|------------------------------|------:|--------|----:|---|-----:|\n|agieval_aqua_rat              |      0|acc     |21.26|±  |  2.57|\n|                              |       |acc_norm|22.83|±  |  2.64|\n|agieval_logiqa_en             |      0|acc     |39.32|±  |  1.92|\n|                              |       |acc_norm|40.71|±  |  1.93|\n|agieval_lsat_ar               |      0|acc     |25.65|±  |  2.89|\n|                              |       |acc_norm|25.65|±  |  2.89|\n|agieval_lsat_lr               |      0|acc     |48.82|±  |  2.22|\n|                              |       |acc_norm|50.00|±  |  2.22|\n|agieval_lsat_rc               |      0|acc     |58.36|±  |  3.01|\n|                              |       |acc_norm|57.25|±  |  3.02|\n|agieval_sat_en                |      0|acc     |74.27|±  |  3.05|\n|                              |       |acc_norm|73.30|±  |  3.09|\n|agieval_sat_en_without_passage|      0|acc     |43.69|±  |  3.46|\n|                              |       |acc_norm|42.23|±  |  3.45|\n|agieval_sat_math              |      0|acc     |37.27|±  |  3.27|\n|                              |       |acc_norm|36.36|±  |  3.25|\n\nAverage: 43.54%\n\n### GPT4All\n|    Task     |Version| Metric |Value|   |Stderr|\n|-------------|------:|--------|----:|---|-----:|\n|arc_challenge|      0|acc     |57.76|±  |  1.44|\n|             |       |acc_norm|60.32|±  |  1.43|\n|arc_easy     |      0|acc     |83.84|±  |  0.76|\n|             |       |acc_norm|81.10|±  |  0.80|\n|boolq        |      1|acc     |86.70|±  |  0.59|\n|hellaswag    |      0|acc     |63.15|±  |  0.48|\n|             |       |acc_norm|82.55|±  |  0.38|\n|openbookqa   |      0|acc     |34.40|±  |  2.13|\n|             |       |acc_norm|45.20|±  |  2.23|\n|piqa         |      0|acc     |81.94|±  |  0.90|\n|             |       |acc_norm|82.97|±  |  0.88|\n|winogrande   |      0|acc     |75.22|±  |  1.21|\n\nAverage: 73.44%\n\n### TruthfulQA\n|    Task     |Version|Metric|Value|   |Stderr|\n|-------------|------:|------|----:|---|-----:|\n|truthfulqa_mc|      1|mc1   |37.70|±  |  1.70|\n|             |       |mc2   |55.26|±  |  1.52|\n\nAverage: 55.26%\n\n### Bigbench\n|                      Task                      |Version|       Metric        |Value|   |Stderr|\n|------------------------------------------------|------:|---------------------|----:|---|-----:|\n|bigbench_causal_judgement                       |      0|multiple_choice_grade|53.16|±  |  3.63|\n|bigbench_date_understanding                     |      0|multiple_choice_grade|65.31|±  |  2.48|\n|bigbench_disambiguation_qa                      |      0|multiple_choice_grade|34.11|±  |  2.96|\n|bigbench_geometric_shapes                       |      0|multiple_choice_grade|27.02|±  |  2.35|\n|                                                |       |exact_str_match      | 0.28|±  |  0.28|\n|bigbench_logical_deduction_five_objects         |      0|multiple_choice_grade|27.80|±  |  2.01|\n|bigbench_logical_deduction_seven_objects        |      0|multiple_choice_grade|19.86|±  |  1.51|\n|bigbench_logical_deduction_three_objects        |      0|multiple_choice_grade|48.33|±  |  2.89|\n|bigbench_movie_recommendation                   |      0|multiple_choice_grade|41.40|±  |  2.20|\n|bigbench_navigate                               |      0|multiple_choice_grade|50.00|±  |  1.58|\n|bigbench_reasoning_about_colored_objects        |      0|multiple_choice_grade|65.00|±  |  1.07|\n|bigbench_ruin_names                             |      0|multiple_choice_grade|46.21|±  |  2.36|\n|bigbench_salient_translation_error_detection    |      0|multiple_choice_grade|27.25|±  |  1.41|\n|bigbench_snarks                                 |      0|multiple_choice_grade|70.72|±  |  3.39|\n|bigbench_sports_understanding                   |      0|multiple_choice_grade|65.72|±  |  1.51|\n|bigbench_temporal_sequences                     |      0|multiple_choice_grade|30.40|±  |  1.46|\n|bigbench_tracking_shuffled_objects_five_objects |      0|multiple_choice_grade|22.56|±  |  1.18|\n|bigbench_tracking_shuffled_objects_seven_objects|      0|multiple_choice_grade|17.09|±  |  0.90|\n|bigbench_tracking_shuffled_objects_three_objects|      0|multiple_choice_grade|48.33|±  |  2.89|\n\nAverage: 42.24%\n\nAverage score: 53.62%\n\n## Usage\n\nYou can run this model using [LM Studio](https://lmstudio.ai/) or any other frontend.\n\nYou can also run this model using the following code:\n\n```python\nimport transformers\nfrom transformers import AutoTokenizer\n\n# Format prompt\nmessage = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant chatbot.\"},\n    {\"role\": \"user\", \"content\": \"What is a Large Language Model?\"}\n]\ntokenizer = AutoTokenizer.from_pretrained(new_model)\nprompt = tokenizer.apply_chat_template(message, add_generation_prompt=True, tokenize=False)\n\n# Create pipeline\npipeline = transformers.pipeline(\n    \"text-generation\",\n    model=\"mlabonne/NeuralHermes-2.5-Mistral-7B-laser\",\n    tokenizer=tokenizer\n)\n\n# Generate text\nsequences = pipeline(\n    prompt,\n    do_sample=True,\n    temperature=0.7,\n    top_p=0.9,\n    num_return_sequences=1,\n    max_length=200,\n)\nprint(sequences[0]['generated_text'])\n```\n# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)\nDetailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_mlabonne__NeuralHermes-2.5-Mistral-7B-laser)\n\n|             Metric              |Value|\n|---------------------------------|----:|\n|Avg.                             |67.29|\n|AI2 Reasoning Challenge (25-Shot)|66.38|\n|HellaSwag (10-Shot)              |85.09|\n|MMLU (5-Shot)                    |63.43|\n|TruthfulQA (0-shot)              |54.95|\n|Winogrande (5-shot)              |78.14|\n|GSM8k (5-shot)                   |55.72|\n\n\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2312.13558",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 100,
  "gated": false,
  "private": false,
  "last_modified": "2024-08-18T05:56:26.000Z",
  "created_at": "2024-08-18T04:45:44.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66c17c78ae70890c90b84950",
  "id": "RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf",
  "modelId": "RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf",
  "sha": "a082aaec63bdad05911ec5864e10341a96a156c5",
  "createdAt": "2024-08-18T04:45:44.000Z",
  "lastModified": "2024-08-18T05:56:26.000Z",
  "author": "RichardErkhov",
  "downloads": 100,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}