---

license: cc-by-nc-4.0

base_model: OpenLLM-Ro/RoLlama3-8b-Instruct

tags:

- gguf

- llama

- llama-3

- romanian

- quantized

- llama.cpp

language:

- ro

- en

pipeline_tag: text-generation

library_name: gguf

---

RoLlama3-8b-Instruct-GGUF

GGUF quantizations of OpenLLM-Ro/RoLlama3-8b-Instruct, converted with llama.cpp.

Original Model

RoLlama3-8b-Instruct is an instruction-tuned Romanian language model from the OpenLLM-Ro project, based on Meta Llama 3 8B. It is fine-tuned for Romanian instruction following and chat.

Features:

• 8B parameters, Llama 3 architecture
• Optimized for Romanian, retains English capability
• 8K context length
• Llama 3 chat template (<|start_header_id|> / <|eot_id|>)

Quantizations

| File | Bits | Approx. Size | Use Case |

|---|---|---|---|

| model_f16.gguf | 16 | ~16.1 GB | Full precision, reference / requantization |

| model_q8_0.gguf | 8 | ~8.5 GB | Near-lossless |

| model_q6_k.gguf | 6 | ~6.6 GB | Very high quality |

| model_q5_k_m.gguf | 5 | ~5.7 GB | High quality, recommended for quality |

| model_q5_k_s.gguf | 5 | ~5.6 GB | High quality, slightly smaller |

| model_q4_k_m.gguf | 4 | ~4.9 GB | Best balance, recommended |

| model_q4_k_s.gguf | 4 | ~4.7 GB | Balanced, smaller |

| model_q3_k_l.gguf | 3 | ~4.3 GB | Low memory, some quality loss |

| model_q3_k_m.gguf | 3 | ~4.0 GB | Low memory |

| model_q3_k_s.gguf | 3 | ~3.7 GB | Very low memory |

| model_q2_k.gguf | 2 | ~3.2 GB | Minimum size, noticeable quality loss |

Memory Requirements

| Quant | RAM / VRAM (approx., 4K context) |

|---|---|

| F16 | ~18 GB |

| Q8_0 | ~10 GB |

| Q6_K | ~8 GB |

| Q5_K_M / Q5_K_S | ~7 GB |

| Q4_K_M / Q4_K_S | ~6 GB |

| Q3_K_L / Q3_K_M / Q3_K_S | ~5-6 GB |

| Q2_K | ~4.5 GB |

Usage

llama.cpp

./llama-cli -m model_q4_k_m.gguf -p "Salut! Cum te poți ajuta?" -n 256 -c 4096

Server mode:

./llama-server -m model_q4_k_m.gguf -c 4096 --port 8080

llama-cpp-python

from llama_cpp import Llama

llm = Llama(model_path="model_q4_k_m.gguf", n_ctx=4096)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Salut! Cum te poți ajuta?"}]
)
print(out["choices"][0]["message"]["content"])

LM Studio

1. Download a .gguf file from this repository.
2. Place it in the LM Studio models folder (<publisher>/<model>/), or use the in-app import.
3. Select the model in the Chat tab and choose the Llama 3 preset.

Ollama

huggingface-cli download tinyopsec/RoLlama3-8b-Instruct-GGUF model_q4_k_m.gguf --local-dir .
cat > Modelfile <<EOF
FROM ./model_q4_k_m.gguf
EOF
ollama create rollama3-8b-instruct -f Modelfile
ollama run rollama3-8b-instruct

Notes

• The extra <|eom_id|> token (id 128256) was removed from the tokenizer before conversion to match the model's embedding size (128256).
• Quantized with llama-quantize from the llama.cpp project.

Credits

• Original model: OpenLLM-Ro/RoLlama3-8b-Instruct
• Base model: Meta Llama 3 8B
• Quantization: tinyopsec