legraphista/aya-23-8b-imat-gguf overview

{
  "metadata": {},
  "card_data": {
    "base_model": "CohereForAI/aya-23-8B",
    "inference": false,
    "language": [
      "en",
      "fr",
      "de",
      "es",
      "it",
      "pt",
      "ja",
      "ko",
      "zh",
      "ar",
      "el",
      "fa",
      "pl",
      "id",
      "cs",
      "he",
      "hi",
      "nl",
      "ro",
      "ru",
      "tr",
      "uk",
      "vi"
    ],
    "library_name": "gguf",
    "license": "cc-by-nc-4.0",
    "pipeline_tag": "text-generation",
    "quantized_by": "legraphista",
    "tags": [
      "quantized",
      "GGUF",
      "imatrix",
      "quantization",
      "imat",
      "static"
    ],
    "frontmatter": {
      "base_model": "CohereForAI/aya-23-8B",
      "inference": "false",
      "language": [
        "en",
        "fr",
        "de",
        "es",
        "it",
        "pt",
        "ja",
        "ko",
        "zh",
        "ar",
        "el",
        "fa",
        "pl",
        "id",
        "cs",
        "he",
        "hi",
        "nl",
        "ro",
        "ru",
        "tr",
        "uk",
        "vi"
      ],
      "library_name": "gguf",
      "license": "cc-by-nc-4.0",
      "pipeline_tag": "text-generation",
      "quantized_by": "legraphista",
      "tags": [
        "quantized",
        "GGUF",
        "imatrix",
        "quantization",
        "imat",
        "static"
      ]
    },
    "hero_image_url": "",
    "summary": "_Llama.cpp imatrix quantization of CohereForAI/aya-23-8B_ Original Model: CohereForAI/aya-23-8B Original dtype: FP16 (float16) Quantized by: llama.cpp b2998 IMatrix dataset: here",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nbase_model: CohereForAI/aya-23-8B\ninference: false\nlanguage:\n- en\n- fr\n- de\n- es\n- it\n- pt\n- ja\n- ko\n- zh\n- ar\n- el\n- fa\n- pl\n- id\n- cs\n- he\n- hi\n- nl\n- ro\n- ru\n- tr\n- uk\n- vi\nlibrary_name: gguf\nlicense: cc-by-nc-4.0\npipeline_tag: text-generation\nquantized_by: legraphista\ntags:\n- quantized\n- GGUF\n- imatrix\n- quantization\n- imat\n- static\n---\n\n# aya-23-8B-IMat-GGUF\n_Llama.cpp imatrix quantization of CohereForAI/aya-23-8B_\n\nOriginal Model: [CohereForAI/aya-23-8B](https://huggingface.co/CohereForAI/aya-23-8B)  \nOriginal dtype: `FP16` (`float16`)  \nQuantized by: llama.cpp [b2998](https://github.com/ggerganov/llama.cpp/releases/tag/b2998)  \nIMatrix dataset: [here](https://gist.githubusercontent.com/legraphista/d6d93f1a254bcfc58e0af3777eaec41e/raw/d380e7002cea4a51c33fffd47db851942754e7cc/imatrix.calibration.medium.raw)  \n\n## Files\n\n### IMatrix\nStatus: ✅ Available  \nLink: [here](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/imatrix.dat) \n\n### Common Quants\n| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |\n| -------- | ---------- | --------- | ------ | ------------ | -------- |\n| [aya-23-8B.Q8_0.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q8_0.gguf) | Q8_0 | 8.54GB | ✅ Available | ⚪ No | 📦 No\n| [aya-23-8B.Q6_K.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q6_K.gguf) | Q6_K | 6.60GB | ✅ Available | ⚪ No | 📦 No\n| [aya-23-8B.Q4_K.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q4_K.gguf) | Q4_K | 5.06GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.Q3_K.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q3_K.gguf) | Q3_K | 4.22GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.Q2_K.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q2_K.gguf) | Q2_K | 3.44GB | ✅ Available | 🟢 Yes | 📦 No\n\n\n### All Quants\n| Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |\n| -------- | ---------- | --------- | ------ | ------------ | -------- |\n| [aya-23-8B.FP16.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.FP16.gguf) | F16 | 16.07GB | ✅ Available | ⚪ No | 📦 No\n| [aya-23-8B.Q5_K.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q5_K.gguf) | Q5_K | 5.80GB | ✅ Available | ⚪ No | 📦 No\n| [aya-23-8B.Q5_K_S.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q5_K_S.gguf) | Q5_K_S | 5.67GB | ✅ Available | ⚪ No | 📦 No\n| [aya-23-8B.Q4_K_S.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q4_K_S.gguf) | Q4_K_S | 4.83GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.Q3_K_L.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q3_K_L.gguf) | Q3_K_L | 4.53GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.Q3_K_S.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q3_K_S.gguf) | Q3_K_S | 3.87GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.Q2_K_S.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.Q2_K_S.gguf) | Q2_K_S | 3.25GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ4_NL.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ4_NL.gguf) | IQ4_NL | 4.81GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ4_XS.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ4_XS.gguf) | IQ4_XS | 4.60GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ3_M.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ3_M.gguf) | IQ3_M | 3.99GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ3_S.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ3_S.gguf) | IQ3_S | 3.89GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ3_XS.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ3_XS.gguf) | IQ3_XS | 3.72GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ3_XXS.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ3_XXS.gguf) | IQ3_XXS | 3.41GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ2_M.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ2_M.gguf) | IQ2_M | 3.08GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ2_S.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ2_S.gguf) | IQ2_S | 2.90GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ2_XS.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ2_XS.gguf) | IQ2_XS | 2.80GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ2_XXS.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ2_XXS.gguf) | IQ2_XXS | 2.59GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ1_M.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ1_M.gguf) | IQ1_M | 2.35GB | ✅ Available | 🟢 Yes | 📦 No\n| [aya-23-8B.IQ1_S.gguf](https://huggingface.co/legraphista/aya-23-8B-IMat-GGUF/blob/main/aya-23-8B.IQ1_S.gguf) | IQ1_S | 2.21GB | ✅ Available | 🟢 Yes | 📦 No\n\n\n## Downloading using huggingface-cli\nFirst, make sure you have hugginface-cli installed:\n```\npip install -U \"huggingface_hub[cli]\"\n```\nThen, you can target the specific file you want:\n```\nhuggingface-cli download legraphista/aya-23-8B-IMat-GGUF --include \"aya-23-8B.Q8_0.gguf\" --local-dir ./\n```\nIf the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:\n```\nhuggingface-cli download legraphista/aya-23-8B-IMat-GGUF --include \"aya-23-8B.Q8_0/*\" --local-dir aya-23-8B.Q8_0\n# see FAQ for merging GGUF's\n```\n\n## FAQ\n\n### Why is the IMatrix not applied everywhere?\nAccording to [this investigation](https://www.reddit.com/r/LocalLLaMA/comments/1993iro/ggufs_quants_can_punch_above_their_weights_now/), it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results). \n\n### How do I merge a split GGUF?\n1. Make sure you have `gguf-split` available\n    - To get hold of `gguf-split`, navigate to https://github.com/ggerganov/llama.cpp/releases\n    - Download the appropriate zip for your system from the latest release\n    - Unzip the archive and you should be able to find `gguf-split`\n2. Locate your GGUF chunks folder (ex: `aya-23-8B.Q8_0`)\n3. Run `gguf-split --merge aya-23-8B.Q8_0/aya-23-8B.Q8_0-00001-of-XXXXX.gguf aya-23-8B.Q8_0.gguf`\n    - Make sure to point `gguf-split` to the first chunk of the split.\n\n---\n\nGot a suggestion? Ping me [@legraphista](https://x.com/legraphista)!",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "quantized",
    "GGUF",
    "imatrix",
    "quantization",
    "imat",
    "static",
    "text-generation",
    "en",
    "fr",
    "de",
    "es",
    "it",
    "pt",
    "ja",
    "ko",
    "zh",
    "ar",
    "el",
    "fa",
    "pl",
    "id",
    "cs",
    "he",
    "hi",
    "nl",
    "ro",
    "ru",
    "tr",
    "uk",
    "vi",
    "base_model:CohereLabs/aya-23-8B",
    "base_model:quantized:CohereLabs/aya-23-8B",
    "license:cc-by-nc-4.0",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 1350,
  "gated": false,
  "private": false,
  "last_modified": "2024-05-26T00:17:38.000Z",
  "created_at": "2024-05-25T20:21:19.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "gguf"
}

{
  "_id": "6652483f95bc4a27ba7b3693",
  "id": "legraphista/aya-23-8B-IMat-GGUF",
  "modelId": "legraphista/aya-23-8B-IMat-GGUF",
  "sha": "e920046538d81bc2d2f2c3e64c5c143e389ae009",
  "createdAt": "2024-05-25T20:21:19.000Z",
  "lastModified": "2024-05-26T00:17:38.000Z",
  "author": "legraphista",
  "downloads": 1350,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "gguf",
  "siblings_count": 29
}