richarderkhov/mistralai_-_mistral-7b-instruct-v0.3-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "The Mistral-7B-Instruct-v0.3 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.3. Mistral-7B-v0.3 has the following changes compared to Mistral-7B-v0.2",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nMistral-7B-Instruct-v0.3 - GGUF\n- Model creator: https://huggingface.co/mistralai/\n- Original model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Mistral-7B-Instruct-v0.3.Q2_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q2_K.gguf) | Q2_K | 2.54GB |\n| [Mistral-7B-Instruct-v0.3.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.IQ3_XS.gguf) | IQ3_XS | 2.82GB |\n| [Mistral-7B-Instruct-v0.3.IQ3_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.IQ3_S.gguf) | IQ3_S | 2.97GB |\n| [Mistral-7B-Instruct-v0.3.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q3_K_S.gguf) | Q3_K_S | 2.95GB |\n| [Mistral-7B-Instruct-v0.3.IQ3_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.IQ3_M.gguf) | IQ3_M | 3.06GB |\n| [Mistral-7B-Instruct-v0.3.Q3_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q3_K.gguf) | Q3_K | 3.28GB |\n| [Mistral-7B-Instruct-v0.3.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q3_K_M.gguf) | Q3_K_M | 3.28GB |\n| [Mistral-7B-Instruct-v0.3.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q3_K_L.gguf) | Q3_K_L | 3.56GB |\n| [Mistral-7B-Instruct-v0.3.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.IQ4_XS.gguf) | IQ4_XS | 3.68GB |\n| [Mistral-7B-Instruct-v0.3.Q4_0.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q4_0.gguf) | Q4_0 | 3.83GB |\n| [Mistral-7B-Instruct-v0.3.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.IQ4_NL.gguf) | IQ4_NL | 3.87GB |\n| [Mistral-7B-Instruct-v0.3.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q4_K_S.gguf) | Q4_K_S | 3.86GB |\n| [Mistral-7B-Instruct-v0.3.Q4_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q4_K.gguf) | Q4_K | 4.07GB |\n| [Mistral-7B-Instruct-v0.3.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf) | Q4_K_M | 4.07GB |\n| [Mistral-7B-Instruct-v0.3.Q4_1.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q4_1.gguf) | Q4_1 | 4.24GB |\n| [Mistral-7B-Instruct-v0.3.Q5_0.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q5_0.gguf) | Q5_0 | 4.66GB |\n| [Mistral-7B-Instruct-v0.3.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q5_K_S.gguf) | Q5_K_S | 4.66GB |\n| [Mistral-7B-Instruct-v0.3.Q5_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q5_K.gguf) | Q5_K | 4.78GB |\n| [Mistral-7B-Instruct-v0.3.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q5_K_M.gguf) | Q5_K_M | 4.78GB |\n| [Mistral-7B-Instruct-v0.3.Q5_1.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q5_1.gguf) | Q5_1 | 5.07GB |\n| [Mistral-7B-Instruct-v0.3.Q6_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q6_K.gguf) | Q6_K | 5.54GB |\n| [Mistral-7B-Instruct-v0.3.Q8_0.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf/blob/main/Mistral-7B-Instruct-v0.3.Q8_0.gguf) | Q8_0 | 7.17GB |\n\n\n\n\nOriginal model description:\n---\nlicense: apache-2.0\n---\n\n# Model Card for Mistral-7B-Instruct-v0.3\n\nThe Mistral-7B-Instruct-v0.3 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.3.\n\nMistral-7B-v0.3 has the following changes compared to [Mistral-7B-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/edit/main/README.md)\n- Extended vocabulary to 32768\n- Supports v3 Tokenizer\n- Supports function calling\n\n## Installation\n\nIt is recommended to use `mistralai/Mistral-7B-Instruct-v0.3` with [mistral-inference](https://github.com/mistralai/mistral-inference). For HF transformers code snippets, please keep scrolling.\n\n```\npip install mistral_inference\n```\n\n## Download\n\n```py\nfrom huggingface_hub import snapshot_download\nfrom pathlib import Path\n\nmistral_models_path = Path.home().joinpath('mistral_models', '7B-Instruct-v0.3')\nmistral_models_path.mkdir(parents=True, exist_ok=True)\n\nsnapshot_download(repo_id=\"mistralai/Mistral-7B-Instruct-v0.3\", allow_patterns=[\"params.json\", \"consolidated.safetensors\", \"tokenizer.model.v3\"], local_dir=mistral_models_path)\n```\n\n### Chat\n\nAfter installing `mistral_inference`, a `mistral-chat` CLI command should be available in your environment. You can chat with the model using\n\n```\nmistral-chat $HOME/mistral_models/7B-Instruct-v0.3 --instruct --max_tokens 256\n```\n\n### Instruct following\n\n```py\nfrom mistral_inference.model import Transformer\nfrom mistral_inference.generate import generate\n\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.protocol.instruct.messages import UserMessage\nfrom mistral_common.protocol.instruct.request import ChatCompletionRequest\n\n\ntokenizer = MistralTokenizer.from_file(f\"{mistral_models_path}/tokenizer.model.v3\")\nmodel = Transformer.from_folder(mistral_models_path)\n\ncompletion_request = ChatCompletionRequest(messages=[UserMessage(content=\"Explain Machine Learning to me in a nutshell.\")])\n\ntokens = tokenizer.encode_chat_completion(completion_request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\n\nprint(result)\n```\n\n### Function calling\n\n```py\nfrom mistral_common.protocol.instruct.tool_calls import Function, Tool\nfrom mistral_inference.model import Transformer\nfrom mistral_inference.generate import generate\n\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.protocol.instruct.messages import UserMessage\nfrom mistral_common.protocol.instruct.request import ChatCompletionRequest\n\n\ntokenizer = MistralTokenizer.from_file(f\"{mistral_models_path}/tokenizer.model.v3\")\nmodel = Transformer.from_folder(mistral_models_path)\n\ncompletion_request = ChatCompletionRequest(\n    tools=[\n        Tool(\n            function=Function(\n                name=\"get_current_weather\",\n                description=\"Get the current weather\",\n                parameters={\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"location\": {\n                            \"type\": \"string\",\n                            \"description\": \"The city and state, e.g. San Francisco, CA\",\n                        },\n                        \"format\": {\n                            \"type\": \"string\",\n                            \"enum\": [\"celsius\", \"fahrenheit\"],\n                            \"description\": \"The temperature unit to use. Infer this from the users location.\",\n                        },\n                    },\n                    \"required\": [\"location\", \"format\"],\n                },\n            )\n        )\n    ],\n    messages=[\n        UserMessage(content=\"What's the weather like today in Paris?\"),\n        ],\n)\n\ntokens = tokenizer.encode_chat_completion(completion_request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\n\nprint(result)\n```\n\n## Generate with `transformers`\n\nIf you want to use Hugging Face `transformers` to generate text, you can do something like this.\n\n```py\nfrom transformers import pipeline\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a pirate chatbot who always responds in pirate speak!\"},\n    {\"role\": \"user\", \"content\": \"Who are you?\"},\n]\nchatbot = pipeline(\"text-generation\", model=\"mistralai/Mistral-7B-Instruct-v0.3\")\nchatbot(messages)\n```\n\n## Limitations\n\nThe Mistral 7B Instruct model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance. \nIt does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to\nmake the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.\n\n## The Mistral AI Team\n\nAlbert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Jean-Malo Delignon, Jia Li, Justus Murke, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Nicolas Schuhl, Patrick von Platen, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibaut Lavril, Timothée Lacroix, Théophile Gervet, Thomas Wang, Valera Nemychnikova, William El Sayed, William Marshall\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 369,
  "gated": false,
  "private": false,
  "last_modified": "2024-06-14T13:16:32.000Z",
  "created_at": "2024-06-14T12:28:19.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "666c3763eeca3ee66060c325",
  "id": "RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf",
  "modelId": "RichardErkhov/mistralai_-_Mistral-7B-Instruct-v0.3-gguf",
  "sha": "0908c20d857015ba1e18eace2b98dff79345de78",
  "createdAt": "2024-06-14T12:28:19.000Z",
  "lastModified": "2024-06-14T13:16:32.000Z",
  "author": "RichardErkhov",
  "downloads": 369,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}