---

license: mit

base_model: HuggingFaceH4/mistral-7b-sft-alpha

tags:

- gguf

- mistral

- quantization

- text-generation

language: en

pipeline_tag: text-generation

library_name: gguf

---

Mistral 7B SFT Alpha - GGUF

GGUF quantized versions of HuggingFaceH4/mistral-7b-sft-alpha, a 7 billion parameter instruction-tuned language model fine-tuned from Mistral 7B v0.1 on the UltraChat dataset.

Model Details

• Base Model: mistralai/Mistral-7B-v0.1
• Fine-tuned by: Hugging Face H4
• Training Data: UltraChat (synthetic dialogue dataset)
• Architecture: Mistral (32 layers, 32 attention heads, GQA)
• Context Length: 4096 tokens
• Vocabulary Size: 32000

Quantization

| Filename | Bits | Size | Use Case |

|----------|------|------|----------|

| model_f16.gguf | 16 | ~14 GB | Maximum quality, high VRAM requirement |

| model_q8_0.gguf | 8 | ~7.5 GB | High quality, moderate VRAM |

| model_q6_k.gguf | 6 | ~5.7 GB | Good quality, balanced VRAM |

| model_q5_k_m.gguf | 5 | ~4.8 GB | Recommended for most use cases |

| model_q5_k_s.gguf | 5 | ~4.2 GB | Compact, minimal quality loss |

| model_q4_k_m.gguf | 4 | ~3.5 GB | Good balance, low VRAM |

| model_q4_k_s.gguf | 4 | ~3.2 GB | Compact model |

| model_q3_k_l.gguf | 3 | ~2.8 GB | Very compact |

| model_q3_k_m.gguf | 3 | ~2.5 GB | Minimal size |

| model_q3_k_s.gguf | 3 | ~2.3 GB | Extreme compression |

| model_q2_k.gguf | 2 | ~1.8 GB | Maximum compression |

VRAM Requirements

| Quantization | VRAM | Recommended Device |

|--------------|------|-------------------|

| F16 | 16 GB | High-end GPU |

| Q8_0 | 8 GB | High-end GPU |

| Q6_K | 6 GB | Mid-range GPU |

| Q5_K_M | 5 GB | Mid-range GPU |

| Q4_K_M | 4 GB | Standard GPU / CPU |

| Q3_K_M | 3 GB | Laptop GPU / CPU |

| Q2_K | 2 GB | CPU with sufficient RAM |

Usage

llama.cpp

./main -m model_q5_k_m.gguf -n 256 -p "You are a helpful assistant."

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="model_q5_k_m.gguf",
    n_ctx=4096,
    n_threads=8
)

response = llm("What is machine learning?", max_tokens=256)
print(response["choices"][0]["text"])

LM Studio

Download the quantized GGUF file and load it in LM Studio's model picker.

Ollama

ollama pull tinyopsec/mistral-7b-sft-alpha-gguf:q5_k_m
ollama run tinyopsec/mistral-7b-sft-alpha-gguf:q5_k_m

Model Capabilities

This model excels at:

• Conversational AI: Fine-tuned on dialogue data for natural interactions
• Instruction Following: Responsive to detailed user instructions
• English Text Generation: Optimized for English-language tasks
• Chat Applications: Suitable for chatbot and virtual assistant use cases

License

This quantized version maintains the original model's MIT license. See the original model card for full details.

Disclaimer

These are community quantizations. For official support and documentation, refer to the original model repository and llama.cpp documentation.