richarderkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-orpo-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "Quantization made by Richard Erkhov. Github Discord Request more models gemma-2-2b-jpn-it-abliterated-17-ORPO - GGUF | Name | Quant method | Size | | ---- | ---- | ---- | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q2_K.gguf | Q2_K | 1.15GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_S.gguf | Q3_K_S | 0.56GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K.gguf | Q3_K | 1.36GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_M.gguf | Q3_K_M | 1.36GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_L.gguf | Q3_K_L | 1.44GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.IQ4_XS.gguf | IQ4_XS | 1.47GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_0.gguf | Q4_0 | 1.52GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.IQ4_NL.gguf | IQ4_NL | 1.53GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K_S.gguf | Q4_K_S | 1.53GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K.gguf | Q4_K | 1.59GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K_M.gguf | Q4_K_M | 1.59GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_1.gguf | Q4_1 | 1.64GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_0.gguf | Q5_0 | 1.75GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K_S.gguf | Q5_K_S | 1.75GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K.gguf | Q5_K | 1.79GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K_M.gguf | Q5_K_M | 1.79GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_1.gguf | Q5_1 | 1.87GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q6_K.gguf | Q6_K | 2.0GB | | gemma-2-2b-jpn-it-abliterated-17-ORPO.Q8_0.gguf | Q8_0 | 2.59GB | Original model description: --- base_model: google/gemma-2-2b-jpn-it language: datasets: library_name: transformers license: gemma license_link: https://ai.google.dev/gemma/terms pipeline_tag: text-generation tags: quantized_by: ymcki widget: content: Can you provide ways to eat combinations of bananas and dragonfruits? --- Original model: https://huggingface.co/google/gemma-2-2b-jpn-it",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\ngemma-2-2b-jpn-it-abliterated-17-ORPO - GGUF\n- Model creator: https://huggingface.co/ymcki/\n- Original model: https://huggingface.co/ymcki/gemma-2-2b-jpn-it-abliterated-17-ORPO/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q2_K.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q2_K.gguf) | Q2_K | 1.15GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_S.gguf) | Q3_K_S | 0.56GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K.gguf) | Q3_K | 1.36GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_M.gguf) | Q3_K_M | 1.36GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q3_K_L.gguf) | Q3_K_L | 1.44GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.IQ4_XS.gguf) | IQ4_XS | 1.47GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_0.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_0.gguf) | Q4_0 | 1.52GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.IQ4_NL.gguf) | IQ4_NL | 1.53GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K_S.gguf) | Q4_K_S | 1.53GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K.gguf) | Q4_K | 1.59GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_K_M.gguf) | Q4_K_M | 1.59GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_1.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q4_1.gguf) | Q4_1 | 1.64GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_0.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_0.gguf) | Q5_0 | 1.75GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K_S.gguf) | Q5_K_S | 1.75GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K.gguf) | Q5_K | 1.79GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_K_M.gguf) | Q5_K_M | 1.79GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_1.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q5_1.gguf) | Q5_1 | 1.87GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q6_K.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q6_K.gguf) | Q6_K | 2.0GB |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO.Q8_0.gguf](https://huggingface.co/RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf/blob/main/gemma-2-2b-jpn-it-abliterated-17-ORPO.Q8_0.gguf) | Q8_0 | 2.59GB |\n\n\n\n\nOriginal model description:\n---\nbase_model: google/gemma-2-2b-jpn-it\nlanguage:\n- multilingual\ndatasets:\n  - mlabonne/orpo-dpo-mix-40k\nlibrary_name: transformers\nlicense: gemma\nlicense_link: https://ai.google.dev/gemma/terms\npipeline_tag: text-generation\ntags:\n- nlp\n- code\nquantized_by: ymcki\nwidget:\n- messages:\n  - role: user\n    content: Can you provide ways to eat combinations of bananas and dragonfruits?\n---\n\nOriginal model: https://huggingface.co/google/gemma-2-2b-jpn-it\n\n## Prompt format\n\n```\n<start_of_turn>user\n{prompt}<end_of_turn>\n<start_of_turn>model\n<end_of_turn>\n<start_of_turn>model\n\n```\n\nNote that this model does not support a System prompt.\n\nThis is abliterated model of [google/gemma-2-2b-jpn-it](https://huggingface.co/google/gemma-2-2b-jpn-it) using the \n[method](https://medium.com/@mlabonne/uncensor-any-llm-with-abliteration-d30148b7d43e) \ndescribed by mlabonne.\n\nLayer 17 of the original model was chosen for abliteration.\nI also created another layer 18 and 24 abliterated model for comparison.\n\nORPO fine tuning was performed for four, eight and twelve epoches. Lowest eval\nat the end of the fourth epoch was at 3.72 epoch. Lowest eval_loss at the \nend of the eighth epoch was 7.48 epoch. Lowest eval_loss at the end of the\ntwelve epoch was 11.96 epoch. Checkpoint at 11.96 epoch was chosen to generate this model.\n\n| Epoch | loss | eval_loss | eval_logps/rejected | eval_logps/chosen |\n| ----- | ---- | --------- | ------------------- | ----------------- |\n| 1.00 | 1.2015 | 1.0501 | -1.0451 | -0.7449 |\n| 2.00 | 1.2576 | 1.0145 | -1.1346 | -0.7248 |\n| 3.00 | 0.9310 | 0.9958 | -1.2629 | -0.7332 |\n| 3.72 | 0.7453 | 0.9848 | -1.2205 | -0.7006 |\n| 4.00 | 0.8866 | 0.9857 | -1.2231 | -0.7019 |\n| 5.00 | 0.8696 | 1.0204 | -1.2242 | -0.7523 |\n| 6.00 | 0.9807 | 0.9959 | -1.3093 | -0.7257 |\n| 7.00 | 0.3851 | 0.9687 | -1.3826 | -0.7103 |\n| 7.48 | 1.2072 | 0.9638 | -1.4512 | -0.6959 |\n| 8.00 | 1.4118 | 0.9653 | -1.5047 | -0.6990 |\n| 9.00 | 1.1466 | 1.0070 | -1.6149 | -0.7567 |\n| 10.00 | 1.4646 | 0.9801 | -1.9078 | -0.7207 |\n| 11.00 | 1.8303 | 0.9620 | -2.0278 | -0.7096 |\n| 11.96 | 0.9252 | 0.9372 | -2.0292 | -0.6692 |\n| 12.00 | 1.1489 | 0.9560 | -1.9191 | -0.7226 |\n\nThe fine tuned model is uploaded here to be evaluated by the Open LLM Leaderboard to see if the slightly brain damaged non-ORPO model can be healed. Again, the fine tuning method is also based on one described by [mlabonne](https://towardsdatascience.com/fine-tune-llama-3-with-orpo-56cfab2f9ada) but the input model was read into VRAM by [unsloth](https://github.com/unslothai/unsloth) to allow using the full 40k dataset to run on a single 3090.\n\n## Benchmark (100.0*raw scores only)\n\nClick on the model name go to the raw score json generated by Open LLM Leaderboard.\n\n| Model | Average | IFEval | BHH | Math Lv5 | GPQA | MUSR | MMLU-PRO |\n| ----- | ------- | ------ | ----|--------- | ---- | ---- | -------- |\n| [gemma-2-2b-jpn-it](https://huggingface.co/datasets/open-llm-leaderboard/results/blob/main/google/gemma-2-2b-jpn-it/results_2024-10-15T15-21-39.173019.json) | 30.82 | 54.11 | 41.43 | 0.0 | 27.52 | 37.17 | 24.67 |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO (4 epoches)](https://huggingface.co/datasets/open-llm-leaderboard/results/raw/main/ymcki/gemma-2-2b-jpn-it-abliterated-17-ORPO/results_2024-10-20T02-46-59.069357.json) | 29.99 | 50.94 | 38.59 | 2.87 | 27.43 | 38.23 | 21.86 |\n| [gemma-2-2b-jpn-it-abliterated-17-ORPO (8 epoches)](https://huggingface.co/datasets/open-llm-leaderboard/results/raw/main/ymcki/gemma-2-2b-jpn-it-abliterated-17-ORPO/results_2024-10-24T00-00-00.000000.json) | 29.42 | 48.95 | 38.27 | 3.17 | 26.93 | 37.43 | 21.77 |\n| gemma-2-2b-jpn-it-abliterated-17-ORPO (12 epoches) | TBD | TBD | TBD | TBD | TBD | TBD | TBD |\n| [gemma-2-2b-jpn-it-abliterated-18-ORPO (4 epoches)](https://huggingface.co/datasets/open-llm-leaderboard/results/raw/main/ymcki/gemma-2-2b-jpn-it-abliterated-18-ORPO/results_2024-10-22T04-04-56.385050.json) | 29.94 | 48.97 | 40.18 | 3.02 | 26.17 | 39.42 | 21.85 |\n| [gemma-2-2b-jpn-it-abliterated-17](https://huggingface.co/datasets/open-llm-leaderboard/results/raw/main/ymcki/gemma-2-2b-jpn-it-abliterated-17/results_2024-10-18T15-18-46.821674.json) | 30.29 | 52.65 | 40.46 | 0.0 | 27.18 | 36.90 | 24.55 |\n| [gemma-2-2b-jpn-it-abliterated-18](https://huggingface.co/datasets/open-llm-leaderboard/results/raw/main/ymcki/gemma-2-2b-jpn-it-abliterated-18/results_2024-10-18T15-41-42.399571.json) | 30.61 | 53.02 | 40.96 | 0.0 | 27.35 | 37.30 | 25.05 |\n| [gemma-2-2b-jpn-it-abliterated-24](https://huggingface.co/datasets/open-llm-leaderboard/results/raw/main/ymcki/gemma-2-2b-jpn-it-abliterated-24/results_2024-10-25T16-29-46.542899.json) | 30.61 | 51.37 | 40.77 | 0.0 | 27.77 | 39.02 | 24.73 |\n\nLooks like fine tuning for 8 epoches is still not enough. May need to run more epoches.\n\n## How to run this model\n\n```py\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nimport transformers\nimport torch\n\nmodel_id = \"gemma-2-2b-jpn-it-abliterated-17-ORPO\"\ndtype = torch.bfloat16\n\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"cuda\",\n    torch_dtype=dtype,)\n\nchat = [\n    { \"role\": \"user\", \"content\": \"Write a hello world program\" },\n]\nprompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)\n```\n\n## Downloading using huggingface-cli\n\nFirst, make sure you have hugginface-cli installed:\n\n```\npip install -U \"huggingface_hub[cli]\"\n```\n\nThen, you can target the specific file you want:\n\n```\nhuggingface-cli download ymcki/gemma-2-2b-jpn-it-abliterated-17-ORPO --include \"*\" --local-dir ./\n```\n\n## Credits\n\nThank you mlabonne for describing his fine tuning method.\n\nThanks FullOf_Bad_Ideas from LocalLlama for the suggestion of using unsloth to save VRAM.\n\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 98,
  "gated": false,
  "private": false,
  "last_modified": "2024-10-29T20:06:39.000Z",
  "created_at": "2024-10-29T18:28:18.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "67212942729a18e052a998cd",
  "id": "RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf",
  "modelId": "RichardErkhov/ymcki_-_gemma-2-2b-jpn-it-abliterated-17-ORPO-gguf",
  "sha": "349a67681508b75ba9742ade9d42121a3be003ec",
  "createdAt": "2024-10-29T18:28:18.000Z",
  "lastModified": "2024-10-29T20:06:39.000Z",
  "author": "RichardErkhov",
  "downloads": 98,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 21
}