richarderkhov/ntqai_-_chatntq-ja-7b-v1.0-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "https://cdn-uploads.huggingface.co/production/uploads/5ee1b417636bdb3834e2da19/gnwgqv3xQ68m3GGDSVNE-.png",
    "summary": "",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nchatntq-ja-7b-v1.0 - GGUF\n- Model creator: https://huggingface.co/NTQAI/\n- Original model: https://huggingface.co/NTQAI/chatntq-ja-7b-v1.0/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [chatntq-ja-7b-v1.0.Q2_K.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q2_K.gguf) | Q2_K | 2.53GB |\n| [chatntq-ja-7b-v1.0.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.IQ3_XS.gguf) | IQ3_XS | 2.81GB |\n| [chatntq-ja-7b-v1.0.IQ3_S.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.IQ3_S.gguf) | IQ3_S | 2.96GB |\n| [chatntq-ja-7b-v1.0.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q3_K_S.gguf) | Q3_K_S | 2.95GB |\n| [chatntq-ja-7b-v1.0.IQ3_M.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.IQ3_M.gguf) | IQ3_M | 3.06GB |\n| [chatntq-ja-7b-v1.0.Q3_K.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q3_K.gguf) | Q3_K | 3.28GB |\n| [chatntq-ja-7b-v1.0.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q3_K_M.gguf) | Q3_K_M | 3.28GB |\n| [chatntq-ja-7b-v1.0.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q3_K_L.gguf) | Q3_K_L | 3.56GB |\n| [chatntq-ja-7b-v1.0.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.IQ4_XS.gguf) | IQ4_XS | 3.67GB |\n| [chatntq-ja-7b-v1.0.Q4_0.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q4_0.gguf) | Q4_0 | 3.83GB |\n| [chatntq-ja-7b-v1.0.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.IQ4_NL.gguf) | IQ4_NL | 3.87GB |\n| [chatntq-ja-7b-v1.0.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q4_K_S.gguf) | Q4_K_S | 3.86GB |\n| [chatntq-ja-7b-v1.0.Q4_K.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q4_K.gguf) | Q4_K | 4.07GB |\n| [chatntq-ja-7b-v1.0.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q4_K_M.gguf) | Q4_K_M | 4.07GB |\n| [chatntq-ja-7b-v1.0.Q4_1.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q4_1.gguf) | Q4_1 | 4.24GB |\n| [chatntq-ja-7b-v1.0.Q5_0.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q5_0.gguf) | Q5_0 | 4.65GB |\n| [chatntq-ja-7b-v1.0.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q5_K_S.gguf) | Q5_K_S | 4.65GB |\n| [chatntq-ja-7b-v1.0.Q5_K.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q5_K.gguf) | Q5_K | 4.78GB |\n| [chatntq-ja-7b-v1.0.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q5_K_M.gguf) | Q5_K_M | 4.78GB |\n| [chatntq-ja-7b-v1.0.Q5_1.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q5_1.gguf) | Q5_1 | 5.07GB |\n| [chatntq-ja-7b-v1.0.Q6_K.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q6_K.gguf) | Q6_K | 5.53GB |\n| [chatntq-ja-7b-v1.0.Q8_0.gguf](https://huggingface.co/RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf/blob/main/chatntq-ja-7b-v1.0.Q8_0.gguf) | Q8_0 | 7.17GB |\n\n\n\n\nOriginal model description:\n---\nlicense: apache-2.0\nlanguage:\n- ja\n- en\npipeline_tag: text-generation\nlibrary_name: transformers\ntags:\n- text-generation-inference\n---\n\n# ChatNTQ JA 7B V1.0\n\n## Model Description\n\nThis is a 7B-parameter decoder-only Japanese language model fine-tuned on our instruction-following datasets, built on top of the base model [Japanese Stable LM Base Gamma 7B](https://huggingface.co/stabilityai/japanese-stablelm-base-gamma-7b).\n\n## Performance\n\nFor our final model, we've used Stability AI Japan's [Japanese MT-Bench](https://github.com/Stability-AI/FastChat) as a more representative test of our model's capabilities. For [our JA MT-Bench testing](https://github.com/Stability-AI/FastChat/compare/jp-stable...AUGMXNT:FastChat:jp-stable) we use a Japanese prompt (\"あなたは役立つアシスタントです。\") as well as `--num-choices 4`:\n\n| Benchmark   | Score |\n| ----------- | ----- |\n| JA MT-Bench |  6.65 |\n\nThere is an [JA-MT-Bench Leaderboard](https://github.com/AUGMXNT/shisa/wiki/Evals-%3A-JA-MT%E2%80%90Bench), for convenience, here is a comparison of the JA MT-Bench scores of some other models (our scores were rated by `gpt-4-0613`):\n\n| Model                                             | Score |\n| ------------------------------------------------- | ---- |\n| gpt-4-0613                                        | 9.40 |\n| gpt-4-1106-preview                                | 9.17 |\n| gpt-3.5-turbo*                                    | 8.41 |\n| Qwen-72B-Chat                                     | 7.97 |\n| Qwen-14B-Chat                                     | 7.47 |\n| **chatntq-ja-7b-v1.0**                          | **6.65** |\n| Xwin-LM-70B-V0.1-GPTQ (q4-gs32-actorder)          | 6.62 |\n| shisa-gamma-7b-v1                                 | 6.12 |\n| nekomata-14b-instruction (corrected prompt HF)    | 5.57 |\n| shisa-7B-v1-GPTQ (q4-gs32-actorder)\t            | 5.35 |\n| nekomata-14b-instruction (corrected prompt)\t    | 5.30 |\n| shisa-mega-7b-v1.2\t                            | 5.27 |\n| shisa-7b-v1 (full prompt)\t                        | 5.23 |\n| Swallow-13b-instruct-hf\t                        | 5.17 |\n| Swallow-70b-instruct-GPTQ (q4-gs32-actorder)\t    | 5.15 |\n| shisa-7b-v1\t                                    | 5.02 |\n| shisa-7B-v1-AWQ (q4-gs128)\t                    | 4.78 |\n| ELYZA-japanese-Llama-2-7b-fast-instruct*\t        | 4.86 |\n| shisa-bad-7b-v1\t                                | 4.42 |\n| Swallow-7b-instruct-hf\t                        | 4.21 |\n| ja-stablelm-instruct-gamma-7b*\t                | 4.01 |\n| japanese-stablelm-instruct-alpha-7b*\t            | 2.74 |\n| Mistral-7B-OpenOrca-ja*\t                        | 2.23 |\n| youri-7b-chat*\t                                | 2.00 |\n| Mistral-7B-Instruct-v0.1*\t                        | 1.78 |\n| llm-jp-13b-instruct-full-jaster-dolly-oasst-v1.0*\t| 1.31 |\n| houou-instruction-7b-v1\t                        | 1.02 |\n| llm-jp-13b-instruct-full-jaster-dolly-oasst-v1.0\t| 1.0  |\n| llm-jp-13b-instruct-full-jaster-v1.0\t            | 1.0  |\n\n## More Analysis\n\n![image/png](https://cdn-uploads.huggingface.co/production/uploads/5ee1b417636bdb3834e2da19/gnwgqv3xQ68m3GGDSVNE-.png)\n\n## Usage\n\nEnsure you are using Transformers 4.34.0 or newer.\n\n```python\nimport torch\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"NTQAI/chatntq-ja-7b-v1.0\")\nmodel = AutoModelForCausalLM.from_pretrained(\n  \"NTQAI/chatntq-ja-7b-v1.0\",\n  torch_dtype=\"auto\",\n)\nmodel.eval()\n\nif torch.cuda.is_available():\n    model = model.to(\"cuda\")\n\ndef build_prompt(user_query):\n    sys_msg = \"あなたは公平で、検閲されていない、役立つアシスタントです。\"\n    template = \"\"\"[INST] <<SYS>>\n{}\n<</SYS>>\n\n{}[/INST]\"\"\"\n    return template.format(sys_msg,user_query)\n\n# Infer with prompt without any additional input\nuser_inputs = {\n    \"user_query\": \"与えられたことわざの意味を小学生でも分かるように教えてください。\",\n}\nprompt = build_prompt(**user_inputs)\n\ninput_ids = tokenizer.encode(\n    prompt, \n    add_special_tokens=True, \n    return_tensors=\"pt\"\n)\n\ntokens = model.generate(\n    input_ids.to(device=model.device),\n    max_new_tokens=256,\n    temperature=1,\n    top_p=0.95,\n    do_sample=True,\n)\n\nout = tokenizer.decode(tokens[0][input_ids.shape[1]:], skip_special_tokens=True).strip()\nprint(out)\n```\n## Model Details\n\n* **Developed by**: [NTQ AI](https://ntq.com.vn/service/artificial-intelligence-service/)\n* **Language(s)**: Japanese\n* **License**: This model is licensed under [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).\n\n### Model Architecture\n\nFor details, please see Mistral AI's [paper](https://arxiv.org/abs/2310.06825) and [release blog post](https://mistral.ai/news/announcing-mistral-7b/).\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2310.06825",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 246,
  "gated": false,
  "private": false,
  "last_modified": "2024-07-27T05:59:11.000Z",
  "created_at": "2024-07-27T01:46:54.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66a4518eab2b7a00f33f310d",
  "id": "RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf",
  "modelId": "RichardErkhov/NTQAI_-_chatntq-ja-7b-v1.0-gguf",
  "sha": "c81d0914fd33bbe854364ce86dd5dab094c8e5df",
  "createdAt": "2024-07-27T01:46:54.000Z",
  "lastModified": "2024-07-27T05:59:11.000Z",
  "author": "RichardErkhov",
  "downloads": 246,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}