richarderkhov/mistralai_-_codestral-22b-v0.1-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nCodestral-22B-v0.1 - GGUF\n- Model creator: https://huggingface.co/mistralai/\n- Original model: https://huggingface.co/mistralai/Codestral-22B-v0.1/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Codestral-22B-v0.1.Q2_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q2_K.gguf) | Q2_K | 7.7GB |\n| [Codestral-22B-v0.1.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.IQ3_XS.gguf) | IQ3_XS | 8.55GB |\n| [Codestral-22B-v0.1.IQ3_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.IQ3_S.gguf) | IQ3_S | 9.02GB |\n| [Codestral-22B-v0.1.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q3_K_S.gguf) | Q3_K_S | 8.98GB |\n| [Codestral-22B-v0.1.IQ3_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.IQ3_M.gguf) | IQ3_M | 9.37GB |\n| [Codestral-22B-v0.1.Q3_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q3_K.gguf) | Q3_K | 10.02GB |\n| [Codestral-22B-v0.1.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q3_K_M.gguf) | Q3_K_M | 10.02GB |\n| [Codestral-22B-v0.1.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q3_K_L.gguf) | Q3_K_L | 10.92GB |\n| [Codestral-22B-v0.1.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.IQ4_XS.gguf) | IQ4_XS | 11.22GB |\n| [Codestral-22B-v0.1.Q4_0.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q4_0.gguf) | Q4_0 | 11.71GB |\n| [Codestral-22B-v0.1.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.IQ4_NL.gguf) | IQ4_NL | 11.83GB |\n| [Codestral-22B-v0.1.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q4_K_S.gguf) | Q4_K_S | 11.79GB |\n| [Codestral-22B-v0.1.Q4_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q4_K.gguf) | Q4_K | 12.42GB |\n| [Codestral-22B-v0.1.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q4_K_M.gguf) | Q4_K_M | 12.42GB |\n| [Codestral-22B-v0.1.Q4_1.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q4_1.gguf) | Q4_1 | 12.99GB |\n| [Codestral-22B-v0.1.Q5_0.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q5_0.gguf) | Q5_0 | 14.27GB |\n| [Codestral-22B-v0.1.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q5_K_S.gguf) | Q5_K_S | 14.27GB |\n| [Codestral-22B-v0.1.Q5_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q5_K.gguf) | Q5_K | 14.64GB |\n| [Codestral-22B-v0.1.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q5_K_M.gguf) | Q5_K_M | 14.64GB |\n| [Codestral-22B-v0.1.Q5_1.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q5_1.gguf) | Q5_1 | 15.56GB |\n| [Codestral-22B-v0.1.Q6_K.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q6_K.gguf) | Q6_K | 17.0GB |\n| [Codestral-22B-v0.1.Q8_0.gguf](https://huggingface.co/RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf/blob/main/Codestral-22B-v0.1.Q8_0.gguf) | Q8_0 | 22.02GB |\n\n\n\n\nOriginal model description:\n---\nlanguage:\n- code\nlicense: other\ntags:\n- code\ninference: false\nlicense_name: mnpl\nlicense_link: https://mistral.ai/licences/MNPL-0.1.md\n\nextra_gated_description: If you want to learn more about how we process your personal data, please read our <a href=\"https://mistral.ai/terms/\">Privacy Policy</a>.\n---\n\n# Model Card for Codestral-22B-v0.1\n\n\n## Encode and Decode with `mistral_common`\n            \n```py\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.protocol.instruct.messages import UserMessage\nfrom mistral_common.protocol.instruct.request import ChatCompletionRequest\n \nmistral_models_path = \"MISTRAL_MODELS_PATH\"\n \ntokenizer = MistralTokenizer.v3()\n \ncompletion_request = ChatCompletionRequest(messages=[UserMessage(content=\"Explain Machine Learning to me in a nutshell.\")])\n \ntokens = tokenizer.encode_chat_completion(completion_request).tokens\n```\n \n## Inference with `mistral_inference`\n \n ```py\nfrom mistral_inference.transformer import Transformer\nfrom mistral_inference.generate import generate\n \nmodel = Transformer.from_folder(mistral_models_path)\nout_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\n\nresult = tokenizer.decode(out_tokens[0])\n\nprint(result)\n```\n\n## Inference with hugging face `transformers`\n \n```py\nfrom transformers import AutoModelForCausalLM\n \nmodel = AutoModelForCausalLM.from_pretrained(\"mistralai/Codestral-22B-v0.1\")\nmodel.to(\"cuda\")\n \ngenerated_ids = model.generate(tokens, max_new_tokens=1000, do_sample=True)\n\n# decode with mistral tokenizer\nresult = tokenizer.decode(generated_ids[0].tolist())\nprint(result)\n```\n\n> [!TIP]\n> PRs to correct the `transformers` tokenizer so that it gives 1-to-1 the same results as the `mistral_common` reference implementation are very welcome!\n            \n---\n\nCodestral-22B-v0.1 is trained on a diverse dataset of 80+ programming languages, including the most popular ones, such as Python, Java, C, C++, JavaScript, and Bash (more details in the [Blogpost](https://mistral.ai/news/codestral/)). The model can be queried:\n- As instruct, for instance to answer any questions about a code snippet (write documentation, explain, factorize) or to generate code following specific indications\n- As Fill in the Middle (FIM), to predict the middle tokens between a prefix and a suffix (very useful for software development add-ons like in VS Code)\n\n\n## Installation\n\nIt is recommended to use `mistralai/Codestral-22B-v0.1` with [mistral-inference](https://github.com/mistralai/mistral-inference).\n\n```\npip install mistral_inference\n```\n\n## Download\n\n```py\nfrom huggingface_hub import snapshot_download\nfrom pathlib import Path\n\nmistral_models_path = Path.home().joinpath('mistral_models', 'Codestral-22B-v0.1')\nmistral_models_path.mkdir(parents=True, exist_ok=True)\n\nsnapshot_download(repo_id=\"mistralai/Codestral-22B-v0.1\", allow_patterns=[\"params.json\", \"consolidated.safetensors\", \"tokenizer.model.v3\"], local_dir=mistral_models_path)\n```\n\n### Chat\n\nAfter installing `mistral_inference`, a `mistral-chat` CLI command should be available in your environment.\n\n```\nmistral-chat $HOME/mistral_models/Codestral-22B-v0.1 --instruct --max_tokens 256\n```\n\nWill generate an answer to \"Write me a function that computes fibonacci in Rust\" and should give something along the following lines:\n\n```\nSure, here's a simple implementation of a function that computes the Fibonacci sequence in Rust. This function takes an integer `n` as an argument and returns the `n`th Fibonacci number.\n\nfn fibonacci(n: u32) -> u32 {\n    match n {\n        0 => 0,\n        1 => 1,\n        _ => fibonacci(n - 1) + fibonacci(n - 2),\n    }\n}\n\nfn main() {\n    let n = 10;\n    println!(\"The {}th Fibonacci number is: {}\", n, fibonacci(n));\n}\n\nThis function uses recursion to calculate the Fibonacci number. However, it's not the most efficient solution because it performs a lot of redundant calculations. A more efficient solution would use a loop to iteratively calculate the Fibonacci numbers.\n```\n\n\n### Fill-in-the-middle (FIM)\n\nAfter installing `mistral_inference` and running `pip install --upgrade mistral_common` to make sure to have mistral_common>=1.2 installed:\n\n```py\nfrom mistral_inference.transformer import Transformer\nfrom mistral_inference.generate import generate\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.tokens.instruct.request import FIMRequest\n\ntokenizer = MistralTokenizer.v3()\nmodel = Transformer.from_folder(\"~/codestral-22B-240529\")\n\nprefix = \"\"\"def add(\"\"\"\nsuffix = \"\"\"    return sum\"\"\"\n\nrequest = FIMRequest(prompt=prefix, suffix=suffix)\n\ntokens = tokenizer.encode_fim(request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.decode(out_tokens[0])\n\nmiddle = result.split(suffix)[0].strip()\nprint(middle)\n```\n\nShould give something along the following lines:\n\n```\nnum1, num2):\n\n    # Add two numbers\n    sum = num1 + num2\n\n    # return the sum\n```\n\n## Usage with transformers library\n\nThis model is also compatible with `transformers` library, first run `pip install -U transformers` then use the snippet below to quickly get started:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"mistralai/Codestral-22B-v0.1\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\ntext = \"Hello my name is\"\ninputs = tokenizer(text, return_tensors=\"pt\")\n\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0], skip_special_tokens=True))\n```\n\nBy default, transformers will load the model in full precision. Therefore you might be interested to further reduce down the memory requirements to run the model through the optimizations we offer in HF ecosystem.\n\n## Limitations\n\nThe Codestral-22B-v0.1 does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to\nmake the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.\n\n## License\n\nCodestral-22B-v0.1 is released under the `MNLP-0.1` license.\n\n## The Mistral AI Team\n\nAlbert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Henri Roussez, Jean-Malo Delignon, Jia Li, Justus Murke, Kartik Khandelwal, Lawrence Stewart, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Marjorie Janiewicz, Mickael Seznec, Nicolas Schuhl, Patrick von Platen, Romain Sauvestre, Pierre Stock, Sandeep Subramanian, Saurabh Garg, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibaut Lavril, Thibault Schueller, Timothée Lacroix, Théophile Gervet, Thomas Wang, Valera Nemychnikova, Wendy Shang, William El Sayed, William Marshall\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 81,
  "gated": false,
  "private": false,
  "last_modified": "2024-09-17T04:20:47.000Z",
  "created_at": "2024-09-16T18:30:25.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66e87941955eeec85e226f94",
  "id": "RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf",
  "modelId": "RichardErkhov/mistralai_-_Codestral-22B-v0.1-gguf",
  "sha": "1921b7ca534bb23f464067163fa88f5d87dc0f58",
  "createdAt": "2024-09-16T18:30:25.000Z",
  "lastModified": "2024-09-17T04:20:47.000Z",
  "author": "RichardErkhov",
  "downloads": 81,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}