richarderkhov/flammenai_-_flammen16-mistral-7b-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "https://huggingface.co/nbeerbower/flammen13X-mistral-7B/resolve/main/flammen13x.png",
    "summary": "A Mistral 7B LLM built from merging pretrained models and finetuning on Jon Durbin's Truthy DPO set. Flammen specializes in exceptional character roleplay, creative writing, and general intelligence ### Method Finetuned using an A100 on Google Colab. 🙏 Fine-tune a Mistral-7b model with Direct Preference Optimization - Maxime Labonne ### Configuration LoRA, model, and training settings: ``python # LoRA configuration peft_config = LoraConfig( r=16, lora_alpha=16, lora_dropout=0.05, bias=\"none\", task_type=\"CAUSAL_LM\", target_modules=['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj'] ) # Model to fine-tune model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, load_in_4bit=True ) model.config.use_cache = False # Reference model ref_model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, load_in_4bit=True ) # Training arguments training_args = TrainingArguments( per_device_train_batch_size=2, gradient_accumulation_steps=2, gradient_checkpointing=True, learning_rate=2e-5, lr_scheduler_type=\"cosine\", max_steps=420, save_strategy=\"no\", logging_steps=1, output_dir=new_model, optim=\"paged_adamw_32bit\", warmup_steps=100, bf16=True, report_to=\"wandb\", ) # Create DPO trainer dpo_trainer = DPOTrainer( model, ref_model, args=training_args, train_dataset=dataset, tokenizer=tokenizer, peft_config=peft_config, beta=0.1, max_prompt_length=1024, max_length=1536, force_use_ref_model=True ) # Fine-tune model with DPO dpo_trainer.train() ``",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nflammen16-mistral-7B - GGUF\n- Model creator: https://huggingface.co/flammenai/\n- Original model: https://huggingface.co/flammenai/flammen16-mistral-7B/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [flammen16-mistral-7B.Q2_K.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q2_K.gguf) | Q2_K | 2.53GB |\n| [flammen16-mistral-7B.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.IQ3_XS.gguf) | IQ3_XS | 2.81GB |\n| [flammen16-mistral-7B.IQ3_S.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.IQ3_S.gguf) | IQ3_S | 2.96GB |\n| [flammen16-mistral-7B.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q3_K_S.gguf) | Q3_K_S | 2.95GB |\n| [flammen16-mistral-7B.IQ3_M.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.IQ3_M.gguf) | IQ3_M | 3.06GB |\n| [flammen16-mistral-7B.Q3_K.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q3_K.gguf) | Q3_K | 3.28GB |\n| [flammen16-mistral-7B.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q3_K_M.gguf) | Q3_K_M | 3.28GB |\n| [flammen16-mistral-7B.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q3_K_L.gguf) | Q3_K_L | 3.56GB |\n| [flammen16-mistral-7B.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.IQ4_XS.gguf) | IQ4_XS | 3.67GB |\n| [flammen16-mistral-7B.Q4_0.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q4_0.gguf) | Q4_0 | 3.83GB |\n| [flammen16-mistral-7B.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.IQ4_NL.gguf) | IQ4_NL | 3.87GB |\n| [flammen16-mistral-7B.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q4_K_S.gguf) | Q4_K_S | 3.86GB |\n| [flammen16-mistral-7B.Q4_K.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q4_K.gguf) | Q4_K | 4.07GB |\n| [flammen16-mistral-7B.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q4_K_M.gguf) | Q4_K_M | 4.07GB |\n| [flammen16-mistral-7B.Q4_1.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q4_1.gguf) | Q4_1 | 4.24GB |\n| [flammen16-mistral-7B.Q5_0.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q5_0.gguf) | Q5_0 | 4.65GB |\n| [flammen16-mistral-7B.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q5_K_S.gguf) | Q5_K_S | 4.65GB |\n| [flammen16-mistral-7B.Q5_K.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q5_K.gguf) | Q5_K | 4.78GB |\n| [flammen16-mistral-7B.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q5_K_M.gguf) | Q5_K_M | 4.78GB |\n| [flammen16-mistral-7B.Q5_1.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q5_1.gguf) | Q5_1 | 5.07GB |\n| [flammen16-mistral-7B.Q6_K.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q6_K.gguf) | Q6_K | 5.53GB |\n| [flammen16-mistral-7B.Q8_0.gguf](https://huggingface.co/RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf/blob/main/flammen16-mistral-7B.Q8_0.gguf) | Q8_0 | 7.17GB |\n\n\n\n\nOriginal model description:\n---\nlibrary_name: transformers\nlicense: apache-2.0\nbase_model:\n  - nbeerbower/flammen15X-mistral-7B\ndatasets:\n- jondurbin/truthy-dpo-v0.1\n---\n\n![image/png](https://huggingface.co/nbeerbower/flammen13X-mistral-7B/resolve/main/flammen13x.png)\n\n# flammen16-mistral-7B\n\nA Mistral 7B LLM built from merging pretrained models and finetuning on [Jon Durbin](https://huggingface.co/jondurbin)'s [Truthy DPO set](https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1). \nFlammen specializes in exceptional character roleplay, creative writing, and general intelligence\n\n### Method\n\nFinetuned using an A100 on Google Colab. 🙏\n\n[Fine-tune a Mistral-7b model with Direct Preference Optimization](https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac) - [Maxime Labonne](https://huggingface.co/mlabonne)\n\n### Configuration\n\nLoRA, model, and training settings:\n\n```python\n# LoRA configuration\npeft_config = LoraConfig(\n    r=16,\n    lora_alpha=16,\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\",\n    target_modules=['k_proj', 'gate_proj', 'v_proj', 'up_proj', 'q_proj', 'o_proj', 'down_proj']\n)\n\n# Model to fine-tune\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    torch_dtype=torch.bfloat16,\n    load_in_4bit=True\n)\nmodel.config.use_cache = False\n\n# Reference model\nref_model = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    torch_dtype=torch.bfloat16,\n    load_in_4bit=True\n)\n\n# Training arguments\ntraining_args = TrainingArguments(\n    per_device_train_batch_size=2,\n    gradient_accumulation_steps=2,\n    gradient_checkpointing=True,\n    learning_rate=2e-5,\n    lr_scheduler_type=\"cosine\",\n    max_steps=420,\n    save_strategy=\"no\",\n    logging_steps=1,\n    output_dir=new_model,\n    optim=\"paged_adamw_32bit\",\n    warmup_steps=100,\n    bf16=True,\n    report_to=\"wandb\",\n)\n\n# Create DPO trainer\ndpo_trainer = DPOTrainer(\n    model,\n    ref_model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer,\n    peft_config=peft_config,\n    beta=0.1,\n    max_prompt_length=1024,\n    max_length=1536,\n    force_use_ref_model=True\n)\n\n# Fine-tune model with DPO\ndpo_trainer.train()\n```\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 88,
  "gated": false,
  "private": false,
  "last_modified": "2024-09-18T06:58:10.000Z",
  "created_at": "2024-09-18T00:50:50.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66ea23eaffe696289bd4d0b7",
  "id": "RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf",
  "modelId": "RichardErkhov/flammenai_-_flammen16-mistral-7B-gguf",
  "sha": "23ccff9690b623009b45603a37ade629f96b442e",
  "createdAt": "2024-09-18T00:50:50.000Z",
  "lastModified": "2024-09-18T06:58:10.000Z",
  "author": "RichardErkhov",
  "downloads": 88,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}