---

license: mit

base_model: kaist-ai/mistral-orpo-capybara-7k

tags:

- gguf

- quantization

- mistral

- orpo

language: en

pipeline_tag: text-generation

library_name: gguf

---

Mistral-ORPO-Capybara-7k GGUF

GGUF quantizations of kaist-ai/mistral-orpo-capybara-7k.

Quantization Variants

| File | Bits | Size | Use Case |

|------|------|------|----------|

| model_f16.gguf | 16 | ~14 GB | Reference, maximum quality |

| model_q8_0.gguf | 8 | ~7.5 GB | High quality, still large |

| model_q6_k.gguf | 6 | ~5.8 GB | Balanced quality/size |

| model_q5_k_m.gguf | 5 | ~4.8 GB | Good quality, recommended |

| model_q5_k_s.gguf | 5 | ~4.6 GB | Slightly faster Q5 |

| model_q4_k_m.gguf | 4 | ~3.8 GB | Standard quantization |

| model_q4_k_s.gguf | 4 | ~3.6 GB | Slightly faster Q4 |

| model_q3_k_l.gguf | 3 | ~3.0 GB | Small model variant |

| model_q3_k_m.gguf | 3 | ~2.9 GB | Balanced 3-bit |

| model_q3_k_s.gguf | 3 | ~2.7 GB | Fast 3-bit |

| model_q2_k.gguf | 2 | ~2.0 GB | Minimal size, CPU-friendly |

VRAM Requirements

| Quantization | GPU VRAM | RAM (CPU) |

|---|---|---|

| F16 | ~14 GB | ~17 GB |

| Q8_0 | ~8 GB | ~10 GB |

| Q6_K | ~6.5 GB | ~8 GB |

| Q5_K_M | ~5.5 GB | ~7 GB |

| Q4_K_M | ~4.5 GB | ~6 GB |

| Q3_K_M | ~3.5 GB | ~5 GB |

| Q2_K | ~2.5 GB | ~3.5 GB |

Usage

llama.cpp

./main -m model_q5_k_m.gguf -n 256 --chat-template mistral -p "Your prompt here"

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="model_q5_k_m.gguf",
    n_gpu_layers=-1,
    n_ctx=2048,
    chat_format="mistral"
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Your prompt here"}],
    max_tokens=256
)
print(response["choices"][0]["message"]["content"])

LM Studio

1. Download any .gguf file from this repo
2. Open in LM Studio
3. Configure context and parameters
4. Start chat

Ollama

ollama create mistral-orpo -f Modelfile

Modelfile:

FROM model_q5_k_m.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Base Model

kaist-ai/mistral-orpo-capybara-7k — Fine-tuned version of Mistral-7B-v0.1 using ORPO (Odds Ratio Preference Optimization) on the Capybara DPO dataset. Trained for 2.5 hours on four A100s.

Performance

• AlpacaEval 2.0 (LC): 15.9%
• MT-Bench: 7.44
• IFEval Inst-Strict: 58.27%

License

MIT (as per base model)

Citation

@misc{hong2024orpo,
      title={ORPO: Monolithic Preference Optimization without Reference Model},
      author={Jiwoo Hong and Noah Lee and James Thorne},
      year={2024},
      eprint={2403.07691},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}