richarderkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "https://netspresso-research-code-release.s3.us-east-2.amazonaws.com/compressed-llm/st-llama_method.png",
    "summary": "Shortened LLaMA is a depth-pruned version of LLaMA models & variants for efficient text generation.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nst-llama-1-5.5b-taylor - GGUF\n- Model creator: https://huggingface.co/nota-ai/\n- Original model: https://huggingface.co/nota-ai/st-llama-1-5.5b-taylor/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [st-llama-1-5.5b-taylor.Q2_K.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q2_K.gguf) | Q2_K | 1.94GB |\n| [st-llama-1-5.5b-taylor.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.IQ3_XS.gguf) | IQ3_XS | 2.15GB |\n| [st-llama-1-5.5b-taylor.IQ3_S.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.IQ3_S.gguf) | IQ3_S | 2.26GB |\n| [st-llama-1-5.5b-taylor.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q3_K_S.gguf) | Q3_K_S | 2.26GB |\n| [st-llama-1-5.5b-taylor.IQ3_M.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.IQ3_M.gguf) | IQ3_M | 2.38GB |\n| [st-llama-1-5.5b-taylor.Q3_K.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q3_K.gguf) | Q3_K | 2.52GB |\n| [st-llama-1-5.5b-taylor.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q3_K_M.gguf) | Q3_K_M | 2.52GB |\n| [st-llama-1-5.5b-taylor.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q3_K_L.gguf) | Q3_K_L | 0.24GB |\n| [st-llama-1-5.5b-taylor.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.IQ4_XS.gguf) | IQ4_XS | 2.79GB |\n| [st-llama-1-5.5b-taylor.Q4_0.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q4_0.gguf) | Q4_0 | 2.93GB |\n| [st-llama-1-5.5b-taylor.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.IQ4_NL.gguf) | IQ4_NL | 2.94GB |\n| [st-llama-1-5.5b-taylor.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q4_K_S.gguf) | Q4_K_S | 2.95GB |\n| [st-llama-1-5.5b-taylor.Q4_K.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q4_K.gguf) | Q4_K | 3.12GB |\n| [st-llama-1-5.5b-taylor.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q4_K_M.gguf) | Q4_K_M | 3.12GB |\n| [st-llama-1-5.5b-taylor.Q4_1.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q4_1.gguf) | Q4_1 | 3.24GB |\n| [st-llama-1-5.5b-taylor.Q5_0.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q5_0.gguf) | Q5_0 | 3.55GB |\n| [st-llama-1-5.5b-taylor.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q5_K_S.gguf) | Q5_K_S | 3.55GB |\n| [st-llama-1-5.5b-taylor.Q5_K.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q5_K.gguf) | Q5_K | 3.65GB |\n| [st-llama-1-5.5b-taylor.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q5_K_M.gguf) | Q5_K_M | 3.65GB |\n| [st-llama-1-5.5b-taylor.Q5_1.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q5_1.gguf) | Q5_1 | 3.87GB |\n| [st-llama-1-5.5b-taylor.Q6_K.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q6_K.gguf) | Q6_K | 4.22GB |\n| [st-llama-1-5.5b-taylor.Q8_0.gguf](https://huggingface.co/RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf/blob/main/st-llama-1-5.5b-taylor.Q8_0.gguf) | Q8_0 | 5.47GB |\n\n\n\n\nOriginal model description:\n# Shortened LLaMA Model Card\n\nShortened LLaMA is a depth-pruned version of LLaMA models & variants for efficient text generation.\n\n- **Developed by:** [Nota AI](https://www.nota.ai/)\n- **License:** Non-commercial license\n- **Repository:** https://github.com/Nota-NetsPresso/shortened-llm\n- **Paper:** https://arxiv.org/abs/2402.02834\n\n## Compression Method\nAfter identifying unimportant Transformer blocks, we perform one-shot pruning and light LoRA-based retraining.\n<details>\n<summary>\nClick to see a method figure.\n</summary>\n\n<img alt=\"method\" img src=\"https://netspresso-research-code-release.s3.us-east-2.amazonaws.com/compressed-llm/st-llama_method.png\" width=\"100%\">\n\n</details>\n\n## Model Links\n  | Source<br>Model | Pruning<br>Ratio | Pruning<br>Criterion | HF Models<br>Link |\n  |:---:|:---:|:---:|:---:|\n  | LLaMA-1-7B | 20% | PPL | [nota-ai/st-llama-1-5.5b-ppl](https://huggingface.co/nota-ai/st-llama-1-5.5b-ppl) |\n  | LLaMA-1-7B | 20% | Taylor+ | [nota-ai/st-llama-1-5.5b-taylor](https://huggingface.co/nota-ai/st-llama-1-5.5b-taylor) |\n  | Vicuna-v1.3-7B | 20% | PPL | [nota-ai/st-vicuna-v1.3-5.5b-ppl](https://huggingface.co/nota-ai/st-vicuna-v1.3-5.5b-ppl) |\n  | Vicuna-v1.3-7B | 20% | Taylor+ | [nota-ai/st-vicuna-v1.3-5.5b-taylor](https://huggingface.co/nota-ai/st-vicuna-v1.3-5.5b-taylor) |\n  | Vicuna-v1.3-13B | 21% | PPL | [nota-ai/st-vicuna-v1.3-10.5b-ppl](https://huggingface.co/nota-ai/st-vicuna-v1.3-10.5b-ppl) |\n  | Vicuna-v1.3-13B | 21% | Taylor+ | [nota-ai/st-vicuna-v1.3-10.5b-taylor](https://huggingface.co/nota-ai/st-vicuna-v1.3-10.5b-taylor) |\n\n## Zero-shot Performance & Efficiency Results\n- EleutherAI/lm-evaluation-harness version [3326c54](https://github.com/EleutherAI/lm-evaluation-harness/tree/3326c547a733d598b4377e54be96e194861b964c)\n\n<img alt=\"results\" img src=\"https://netspresso-research-code-release.s3.us-east-2.amazonaws.com/compressed-llm/st-llama_zero-shot_scores.png\" width=\"100%\">\n\n## License\n- All rights related to this repository and the compressed models are reserved by Nota Inc.\n- The intended use is strictly limited to research and non-commercial projects.\n\n## Acknowledgments\n- [LLM-Pruner](https://github.com/horseee/LLM-Pruner), which utilizes [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness), [PEFT](https://github.com/huggingface/peft), and [Alpaca-LoRA](https://github.com/tloen/alpaca-lora). Thanks for the pioneering work on structured pruning of LLMs! \n- Meta AI's [LLaMA](https://github.com/facebookresearch/llama) and  LMSYS Org's [Vicuna](https://github.com/lm-sys/FastChat/blob/main/docs/vicuna_weights_version.md). Thanks for the open-source LLMs!\n\n## Citation\n```bibtex\n@article{kim2024shortened,\n  title={Shortened LLaMA: A Simple Depth Pruning for Large Language Models},\n  author={Kim, Bo-Kyeong and Kim, Geonmin and Kim, Tae-Ho and Castells, Thibault and Choi, Shinkook and Shin, Junho and Song, Hyoung-Kyu},\n  journal={arXiv preprint arXiv:2402.02834},      \n  year={2024},\n  url={https://arxiv.org/abs/2402.02834}\n}\n```\n```bibtex\n@article{kim2024mefomo,\n  title={Shortened LLaMA: A Simple Depth Pruning for Large Language Models},\n  author={Kim, Bo-Kyeong and Kim, Geonmin and Kim, Tae-Ho and Castells, Thibault and Choi, Shinkook and Shin, Junho and Song, Hyoung-Kyu},\n  journal={ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo)},\n  year={2024},\n  url={https://openreview.net/forum?id=18VGxuOdpu}\n}\n```\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2402.02834",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 154,
  "gated": false,
  "private": false,
  "last_modified": "2024-10-04T04:00:55.000Z",
  "created_at": "2024-10-03T20:46:48.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66ff02b8001816b29e834157",
  "id": "RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf",
  "modelId": "RichardErkhov/nota-ai_-_st-llama-1-5.5b-taylor-gguf",
  "sha": "96eb61e2986d711e938572d67665102b30982ac9",
  "createdAt": "2024-10-03T20:46:48.000Z",
  "lastModified": "2024-10-04T04:00:55.000Z",
  "author": "RichardErkhov",
  "downloads": 154,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}