richarderkhov/local-novel-llm-project_-_ninja-v1-nsfw-128k-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "./ninjalogo.svg",
    "summary": "",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nNinja-v1-NSFW-128k - GGUF\n- Model creator: https://huggingface.co/Local-Novel-LLM-project/\n- Original model: https://huggingface.co/Local-Novel-LLM-project/Ninja-v1-NSFW-128k/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Ninja-v1-NSFW-128k.Q2_K.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q2_K.gguf) | Q2_K | 2.53GB |\n| [Ninja-v1-NSFW-128k.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q3_K_S.gguf) | Q3_K_S | 2.95GB |\n| [Ninja-v1-NSFW-128k.Q3_K.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q3_K.gguf) | Q3_K | 3.28GB |\n| [Ninja-v1-NSFW-128k.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q3_K_M.gguf) | Q3_K_M | 3.28GB |\n| [Ninja-v1-NSFW-128k.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q3_K_L.gguf) | Q3_K_L | 3.56GB |\n| [Ninja-v1-NSFW-128k.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.IQ4_XS.gguf) | IQ4_XS | 3.67GB |\n| [Ninja-v1-NSFW-128k.Q4_0.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q4_0.gguf) | Q4_0 | 3.83GB |\n| [Ninja-v1-NSFW-128k.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.IQ4_NL.gguf) | IQ4_NL | 3.87GB |\n| [Ninja-v1-NSFW-128k.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q4_K_S.gguf) | Q4_K_S | 3.86GB |\n| [Ninja-v1-NSFW-128k.Q4_K.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q4_K.gguf) | Q4_K | 4.07GB |\n| [Ninja-v1-NSFW-128k.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q4_K_M.gguf) | Q4_K_M | 4.07GB |\n| [Ninja-v1-NSFW-128k.Q4_1.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q4_1.gguf) | Q4_1 | 4.24GB |\n| [Ninja-v1-NSFW-128k.Q5_0.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q5_0.gguf) | Q5_0 | 4.65GB |\n| [Ninja-v1-NSFW-128k.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q5_K_S.gguf) | Q5_K_S | 4.65GB |\n| [Ninja-v1-NSFW-128k.Q5_K.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q5_K.gguf) | Q5_K | 4.78GB |\n| [Ninja-v1-NSFW-128k.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q5_K_M.gguf) | Q5_K_M | 4.78GB |\n| [Ninja-v1-NSFW-128k.Q5_1.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q5_1.gguf) | Q5_1 | 5.07GB |\n| [Ninja-v1-NSFW-128k.Q6_K.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q6_K.gguf) | Q6_K | 5.53GB |\n| [Ninja-v1-NSFW-128k.Q8_0.gguf](https://huggingface.co/RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf/blob/main/Ninja-v1-NSFW-128k.Q8_0.gguf) | Q8_0 | 7.17GB |\n\n\n\n\nOriginal model description:\n---\nlicense: apache-2.0\nlanguage:\n- en\n- ja\ntags:\n- finetuned\n- not-for-all-audiences\nlibrary_name: transformers\npipeline_tag: text-generation\n---\n<img src=\"./ninjalogo.svg\" width=\"100%\" height=\"20%\" alt=\"\"> \n\n# Our Models\n- [Vecteus](https://huggingface.co/Local-Novel-LLM-project/Vecteus-v1)\n\n- [Ninja-v1](https://huggingface.co/Local-Novel-LLM-project/Ninja-v1) \n\n- [Ninja-v1-NSFW](https://huggingface.co/Local-Novel-LLM-project/Ninja-v1-NSFW)\n\n- [Ninja-v1-128k](https://huggingface.co/Local-Novel-LLM-project/Ninja-v1-128k)\n\n- [Ninja-v1-NSFW-128k](https://huggingface.co/Local-Novel-LLM-project/Ninja-v1-NSFW-128k)\n\n## Model Card for Ninja-v1-NSFW-128k\n\nThe Mistral-7B--based Large Language Model (LLM) is an noveldataset fine-tuned version of the Mistral-7B-v0.1\n\nNinja-NSFW-128k has the following changes compared to Mistral-7B-v0.1.\n- 128k context window (8k context in v0.1)\n- Achieving both high quality Japanese and English generation\n- Memory ability that does not forget even after long-context generation\n- Can be generated NSFW\n\nThis model was created with the help of GPUs from the first LocalAI hackathon.\n\nWe would like to take this opportunity to thank\n\n## List of Creation Methods\n\n- Chatvector for multiple models\n- Simple linear merging of result models\n- Domain and Sentence Enhancement with LORA\n- Context expansion\n\n## Instruction format\n\n  Ninja adopts the prompt format from Vicuna and supports multi-turn conversation.\n  The prompt should be as following:\n  ```\n  USER: Hi ASSISTANT: Hello.</s>\n  USER: Who are you?\n  ASSISTANT: I am ninja.</s>\n  ```\n\n## Example prompts to improve (Japanese)\n\n  - BAD:　あなたは○○として振る舞います\n  - GOOD: あなたは○○です\n\n  - BAD: あなたは○○ができます\n  - GOOD: あなたは○○をします\n\n## Performing inference\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\nmodel_id = \"Local-Novel-LLM-project/Ninja-v1-NSFW-128k\"\nnew_tokens = 1024\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\", device_map=\"auto\")\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\nsystem_prompt = \"あなたはプロの小説家です。\\n小説を書いてください\\n-------- \"\n\nprompt = input(\"Enter a prompt: \")\nsystem_prompt += prompt + \"\\n-------- \"\nmodel_inputs = tokenizer([system_prompt], return_tensors=\"pt\").to(\"cuda\")\n\n\ngenerated_ids = model.generate(**model_inputs, max_new_tokens=new_tokens, do_sample=True)\nprint(tokenizer.batch_decode(generated_ids)[0])\n````\n\n## Merge recipe\n\n- WizardLM2 - mistralai/Mistral-7B-v0.1\n- NousResearch/Yarn-Mistral-7b-128k - mistralai/Mistral-7B-v0.1\n- Elizezen/Antler-7B - stabilityai/japanese-stablelm-instruct-gamma-7b\n- Elizezen/LewdSniffyOtter-7B - Elizezen/SniffyOtter-7B\n- NTQAI/chatntq-ja-7b-v1.0\n\nThe characteristics of each model are as follows.\n\n- WizardLM2: High quality multitasking model\n- Yarn-Mistral-7b-128k: Mistral model with 128k context window\n- Antler-7B: Model specialized for novel writing\n- NTQAI/chatntq-ja-7b-v1.0 High quality Japanese specialized model\n- Elizezen/LewdSniffyOtter-7B Japanese NSFW specialized model\n\n## Other points to keep in mind\n- The training data may be biased. Be careful with the generated sentences.\n- Set trust_remote_code to True for context expansion with YaRN.\n- Memory usage may be large for long inferences.\n- If possible, we recommend inferring with llamacpp rather than Transformers.\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 3,
  "downloads": 605,
  "gated": false,
  "private": false,
  "last_modified": "2024-11-13T06:03:04.000Z",
  "created_at": "2024-11-13T01:41:08.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "673403b4db744d70cb84f5f4",
  "id": "RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf",
  "modelId": "RichardErkhov/Local-Novel-LLM-project_-_Ninja-v1-NSFW-128k-gguf",
  "sha": "3ff2c31a6d5b95ce57e3717608b3ddde061caef2",
  "createdAt": "2024-11-13T01:41:08.000Z",
  "lastModified": "2024-11-13T06:03:04.000Z",
  "author": "RichardErkhov",
  "downloads": 605,
  "likes": 3,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 21
}