richarderkhov/patronusai_-_llama-3-patronus-lynx-8b-instruct-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "Lynx is an open-source hallucination evaluation model. Patronus-Lynx-8B-Instruct was trained on a mix of datasets including CovidQA, PubmedQA, DROP, RAGTruth. The datasets contain a mix of hand-annotated and synthetic data. The maximum sequence length is 8000 tokens.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nLlama-3-Patronus-Lynx-8B-Instruct - GGUF\n- Model creator: https://huggingface.co/PatronusAI/\n- Original model: https://huggingface.co/PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q2_K.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q2_K.gguf) | Q2_K | 2.96GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.IQ3_XS.gguf) | IQ3_XS | 3.28GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.IQ3_S.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.IQ3_S.gguf) | IQ3_S | 3.43GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q3_K_S.gguf) | Q3_K_S | 3.41GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.IQ3_M.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.IQ3_M.gguf) | IQ3_M | 3.52GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q3_K.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q3_K.gguf) | Q3_K | 3.74GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q3_K_M.gguf) | Q3_K_M | 3.74GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q3_K_L.gguf) | Q3_K_L | 4.03GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.IQ4_XS.gguf) | IQ4_XS | 4.18GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q4_0.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q4_0.gguf) | Q4_0 | 4.34GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.IQ4_NL.gguf) | IQ4_NL | 4.38GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q4_K_S.gguf) | Q4_K_S | 4.37GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q4_K.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q4_K.gguf) | Q4_K | 4.58GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q4_K_M.gguf) | Q4_K_M | 4.58GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q4_1.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q4_1.gguf) | Q4_1 | 4.78GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q5_0.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q5_0.gguf) | Q5_0 | 5.21GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q5_K_S.gguf) | Q5_K_S | 5.21GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q5_K.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q5_K.gguf) | Q5_K | 5.34GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q5_K_M.gguf) | Q5_K_M | 5.34GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q5_1.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q5_1.gguf) | Q5_1 | 5.65GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q6_K.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q6_K.gguf) | Q6_K | 6.14GB |\n| [Llama-3-Patronus-Lynx-8B-Instruct.Q8_0.gguf](https://huggingface.co/RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf/blob/main/Llama-3-Patronus-Lynx-8B-Instruct.Q8_0.gguf) | Q8_0 | 2.67GB |\n\n\n\n\nOriginal model description:\n---\nlibrary_name: transformers\ntags:\n- text-generation\n- pytorch\n- Lynx\n- Patronus AI\n- evaluation\n- hallucination-detection\nlicense: cc-by-nc-4.0\nlanguage:\n- en\n---\n\n# Model Card for Model ID\n\nLynx is an open-source hallucination evaluation model. Patronus-Lynx-8B-Instruct was trained on a mix of datasets including CovidQA, PubmedQA, DROP, RAGTruth.\nThe datasets contain a mix of hand-annotated and synthetic data. The maximum sequence length is 8000 tokens. \n\n\n## Model Details\n\n- **Model Type:** Patronus-Lynx-8B-Instruct is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct model.\n- **Language:** Primarily English\n- **Developed by:** Patronus AI\n- **Paper:** [https://arxiv.org/abs/2407.08488](https://arxiv.org/abs/2407.08488)\n- **License:** [https://creativecommons.org/licenses/by-nc/4.0/](https://creativecommons.org/licenses/by-nc/4.0/)\n\n### Model Sources\n\n<!-- Provide the basic links for the model. -->\n\n- **Repository:** [https://github.com/patronus-ai/Lynx-hallucination-detection](https://github.com/patronus-ai/Lynx-hallucination-detection)\n\n\n## How to Get Started with the Model\nLynx is trained to detect hallucinations in RAG settings. Provided a document, question and answer, the model can evaluate whether the answer is faithful to the document.\n\nTo use the model, we recommend using the following prompt:\n\n```\nPROMPT = \"\"\"\nGiven the following QUESTION, DOCUMENT and ANSWER you must analyze the provided answer and determine whether it is faithful to the contents of the DOCUMENT. The ANSWER must not offer new information beyond the context provided in the DOCUMENT. The ANSWER also must not contradict information provided in the DOCUMENT. Output your final verdict by strictly following this format: \"PASS\" if the answer is faithful to the DOCUMENT and \"FAIL\" if the answer is not faithful to the DOCUMENT. Show your reasoning.\n\n--\nQUESTION (THIS DOES NOT COUNT AS BACKGROUND INFORMATION):\n{question}\n\n--\nDOCUMENT:\n{context}\n\n--\nANSWER:\n{answer}\n\n--\n\nYour output should be in JSON FORMAT with the keys \"REASONING\" and \"SCORE\":\n{{\"REASONING\": <your reasoning as bullet points>, \"SCORE\": <your final score>}}\n\"\"\"\n```\n\nThe model will output the score as 'PASS' if the answer is faithful to the document or FAIL if the answer is not faithful to the document. \n\n## Inference\n\nTo run inference, you can use HF pipeline:\n\n```\n\nmodel_name = 'PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct'\npipe = pipeline(\n          \"text-generation\",\n          model=model_name,\n          max_new_tokens=600,\n          device=\"cuda\",\n          return_full_text=False\n        )\n\nmessages = [\n    {\"role\": \"user\", \"content\": prompt},\n]\n\nresult = pipe(messages)\nprint(result[0]['generated_text'])\n\n```\n\nSince the model is trained in chat format, ensure that you pass the prompt as a user message.\n\nFor more information on training details, refer to our [ArXiv paper](https://arxiv.org/abs/2407.08488).\n\n## Evaluation\n\nThe model was evaluated on [PatronusAI/HaluBench](https://huggingface.co/datasets/PatronusAI/HaluBench).\n\n\n| Model | HaluEval | RAGTruth | FinanceBench | DROP | CovidQA | PubmedQA | Overall\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| GPT-4o | 87.9% | 84.3% | **85.3%** | 84.3% | 95.0% | 82.1% | 86.5% |\n| GPT-4-Turbo | 86.0% | **85.0%** | 82.2% | 84.8% | 90.6% | 83.5% | 85.0% |\n| GPT-3.5-Turbo | 62.2% | 50.7% | 60.9% | 57.2% | 56.7% | 62.8% | 58.7% |\n| Claude-3-Sonnet | 84.5% | 79.1% | 69.7% | 84.3% | 95.0% | 82.9% | 78.8% |\n| Claude-3-Haiku | 68.9% | 78.9% | 58.4% | 84.3% | 95.0% | 82.9% | 69.0% |\n| RAGAS Faithfulness | 70.6% | 75.8% | 59.5% | 59.6% | 75.0% | 67.7% | 66.9% |\n| Mistral-Instruct-7B | 78.3% | 77.7% | 56.3% | 56.3% | 71.7% | 77.9% | 69.4% |\n| Llama-3-Instruct-8B | 83.1% | 80.0% | 55.0% | 58.2% | 75.2% | 70.7% | 70.4% |\n| Llama-3-Instruct-70B | 87.0% | 83.8% | 72.7% | 69.4% | 85.0% | 82.6% | 80.1% |\n| LYNX (8B) | 85.7% | 80.0% | 72.5% | 77.8% | 96.3% | 85.2% | 82.9% |\n| LYNX (70B) | **88.4%** | 80.2% | 81.4% | **86.4%** | **97.5%** | **90.4%** | **87.4%** |\n\n\n## Citation\nIf you are using the model, cite using\n\n```\n@article{ravi2024lynx,\n  title={Lynx: An Open Source Hallucination Evaluation Model},\n  author={Ravi, Selvan Sunitha and Mielczarek, Bartosz and Kannappan, Anand and Kiela, Douwe and Qian, Rebecca},\n  journal={arXiv preprint arXiv:2407.08488},\n  year={2024}\n}\n```\n\n## Model Card Contact\n[@sunitha-ravi](https://huggingface.co/sunitha-ravi)\n[@RebeccaQian1](https://huggingface.co/RebeccaQian1)\n[@presidev](https://huggingface.co/presidev)\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2407.08488",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 546,
  "gated": false,
  "private": false,
  "last_modified": "2024-08-06T00:08:22.000Z",
  "created_at": "2024-08-05T22:01:23.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66b14bb3874d8abe0b419797",
  "id": "RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf",
  "modelId": "RichardErkhov/PatronusAI_-_Llama-3-Patronus-Lynx-8B-Instruct-gguf",
  "sha": "3dec7453bbad2acc89bc5a0a1184023829138075",
  "createdAt": "2024-08-05T22:01:23.000Z",
  "lastModified": "2024-08-06T00:08:22.000Z",
  "author": "RichardErkhov",
  "downloads": 546,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}