davzoku/cria-llama2-7b-v1.3-gguf overview

{
  "metadata": {},
  "card_data": {
    "inference": false,
    "language": "en",
    "license": "llama2",
    "model_type": "llama",
    "datasets": [
      "mlabonne/CodeLlama-2-20k"
    ],
    "pipeline_tag": "text-generation",
    "tags": [
      "llama-2"
    ],
    "frontmatter": {
      "inference": "false",
      "language": "en",
      "license": "llama2",
      "model_type": "llama",
      "datasets": [
        "mlabonne/CodeLlama-2-20k"
      ],
      "pipeline_tag": "text-generation",
      "tags": [
        "llama-2"
      ]
    },
    "hero_image_url": "https://raw.githubusercontent.com/davzoku/cria/main/assets/icon-512x512.png",
    "summary": "💡 Article | 💻 Github | 📔 Colab 1,2",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\ninference: false\nlanguage: en\nlicense: llama2\nmodel_type: llama\ndatasets:\n  - mlabonne/CodeLlama-2-20k\npipeline_tag: text-generation\ntags:\n  - llama-2\n---\n\n# CRIA v1.3\n\n💡 [Article](https://walterteng.com/cria) |\n💻 [Github](https://github.com/davzoku/cria) |\n📔 Colab [1](https://colab.research.google.com/drive/1rYTs3qWJerrYwihf1j0f00cnzzcpAfYe),[2](https://colab.research.google.com/drive/1Wjs2I1VHjs6zT_GE42iEXsLtYh6VqiJU)\n\n## What is CRIA?\n\n> krē-ə plural crias. : a baby llama, alpaca, vicuña, or guanaco.\n\n<p align=\"center\">\n  <img src=\"https://raw.githubusercontent.com/davzoku/cria/main/assets/icon-512x512.png\" width=\"300\" height=\"300\" alt=\"Cria Logo\"> <br>\n  <i>or what ChatGPT suggests, <b>\"Crafting a Rapid prototype of an Intelligent llm App using open source resources\"</b>.</i>\n</p>\n\nThe initial objective of the CRIA project is to develop a comprehensive end-to-end chatbot system, starting from the instruction-tuning of a large language model and extending to its deployment on the web using frameworks such as Next.js.\n\nSpecifically, we have fine-tuned the `llama-2-7b-chat-hf` model with QLoRA (4-bit precision) using the [mlabonne/CodeLlama-2-20k](https://huggingface.co/datasets/mlabonne/CodeLlama-2-20k) dataset. This fine-tuned model serves as the backbone for the [CRIA chat](https://chat.walterteng.com) platform.\n\n## 📦 Model Release\n\nCRIA v1.3 comes with several variants.\n\n- [davzoku/cria-llama2-7b-v1.3](https://huggingface.co/davzoku/cria-llama2-7b-v1.3): Merged Model\n- [davzoku/cria-llama2-7b-v1.3-GGML](https://huggingface.co/davzoku/cria-llama2-7b-v1.3-GGML): Quantized Merged Model\n- [davzoku/cria-llama2-7b-v1.3_peft](https://huggingface.co/davzoku/cria-llama2-7b-v1.3_peft): PEFT adapter\n- [davzoku/cria-llama2-7b-v1.3-GGUF](https://huggingface.co/davzoku/cria-llama2-7b-v1.3-GGUF): GGUF Format\n  - converted from ggml vq4_0 using `python3 convert-llama-ggml-to-gguf.py -i ../text-generation-webui/models/cria/cria-llama2-7b-v1.3.ggmlv3.q4_0.bin -o cria-llama2-7b-v1.3.gguf`\n\n\nThis model is converted from the q4_0 GGML version of CRIA v1.3 using the [llama.cpp's convert-llama-ggml-to-gguf.py](https://github.com/ggerganov/llama.cpp/commit/f23c0359a32871947169a044eb1dc4dbffd0f405#diff-ca59e5f600b44e9c3cdfed1ffd04677524dfd4c2f43d8b4bb19fdb013e277871) script\n\n## 🔧 Training\n\nIt was trained on a Google Colab notebook with a T4 GPU and high RAM.\n\n### Training procedure\n\nThe following `bitsandbytes` quantization config was used during training:\n\n- load_in_8bit: False\n- load_in_4bit: True\n- llm_int8_threshold: 6.0\n- llm_int8_skip_modules: None\n- llm_int8_enable_fp32_cpu_offload: False\n- llm_int8_has_fp16_weight: False\n- bnb_4bit_quant_type: nf4\n- bnb_4bit_use_double_quant: False\n- bnb_4bit_compute_dtype: float16\n\n### Framework versions\n\n- PEFT 0.4.0\n\n## 💻 Usage\n\n```python\n# pip install transformers accelerate\n\nfrom transformers import AutoTokenizer\nimport transformers\nimport torch\n\nmodel = \"davzoku/cria-llama2-7b-v1.3\"\nprompt = \"What is a cria?\"\n\ntokenizer = AutoTokenizer.from_pretrained(model)\npipeline = transformers.pipeline(\n    \"text-generation\",\n    model=model,\n    torch_dtype=torch.float16,\n    device_map=\"auto\",\n)\n\nsequences = pipeline(\n    f'<s>[INST] {prompt} [/INST]',\n    do_sample=True,\n    top_k=10,\n    num_return_sequences=1,\n    eos_token_id=tokenizer.eos_token_id,\n    max_length=200,\n)\nfor seq in sequences:\n    print(f\"Result: {seq['generated_text']}\")\n```\n\n## References\n\nWe'd like to thank:\n\n- [mlabonne](https://huggingface.co/mlabonne) for his article and resources on implementation of instruction tuning\n- [TheBloke](https://huggingface.co/TheBloke) for his script for LLM quantization.\n",
    "related_quantizations": []
  },
  "tags": [
    "transformers",
    "pytorch",
    "gguf",
    "llama",
    "text-generation",
    "llama-2",
    "en",
    "dataset:mlabonne/CodeLlama-2-20k",
    "license:llama2",
    "text-generation-inference",
    "region:us"
  ],
  "likes": 0,
  "downloads": 527,
  "gated": false,
  "private": false,
  "last_modified": "2024-02-09T04:57:33.000Z",
  "created_at": "2024-01-14T16:37:34.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "transformers"
}

{
  "_id": "65a40dce0699df1be5b5f00a",
  "id": "davzoku/cria-llama2-7b-v1.3-GGUF",
  "modelId": "davzoku/cria-llama2-7b-v1.3-GGUF",
  "sha": "c43f4f1353cd61142c5f7f4bba531a51f54e6d0a",
  "createdAt": "2024-01-14T16:37:34.000Z",
  "lastModified": "2024-02-09T04:57:33.000Z",
  "author": "davzoku",
  "downloads": 527,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "transformers",
  "siblings_count": 12
}