GraySoft
Projects Models About FAQ Contact Download guIDE →
Model Intelligence Sheet

richarderkhov/ilsp_-_meltemi-7b-instruct-v1.5-gguf overview

We present Meltemi 7B Instruct v1.5 Large Language Model (LLM), a new and improved instruction fine-tuned version of Meltemi 7B v1.5. !image/png # Model Information 89,730 Greek preference data which are mostly translated versions of high-quality datasets on Hugging Face 7,342 English preference data # Instruction format The prompt format is the same as the Zephyr format and can be utilized through the tokenizer's chat template functionality as follows: Please make sure that the BOS token is always included in the tokenized prompts. This might not be the default setting in all evaluation or fine-tuning frameworks. # Evaluation The evaluation suite we created includes 6 test sets and has been implemented based on a fork of the lighteval framework. Our evaluation suite includes: Four machine-translated versions (ARC Greek, Truthful QA Greek, HellaSwag Greek, MMLU Greek) of established English benchmarks for language understanding and reasoning (ARC Challenge, Truthful QA, Hellaswag, MMLU). An existing benchmark for question answering in Greek (Belebele) * A novel benchmark created by the ILSP team for medical question answering based on the medical exams of DOATAP (Medical MCQA). Our evaluation is performed in a few-shot setting, consistent with the settings in the Open LLM leaderboard. We can see that our new training and fine-tuning procedure for Meltemi 7B Instruct v1.5 enhances performance across all Greek test sets by a +7.8% average improvement compared to the earlier Meltemi Instruct 7B v1 model. The results for the Greek test sets are shown in the following table: | | Medical MCQA EL (15-shot) | Belebele EL (5-shot) | HellaSwag EL (10-shot) | ARC-Challenge EL (25-shot) | TruthfulQA MC2 EL (0-shot) | MMLU EL (5-shot) | Average | |----------------|----------------|-------------|--------------|------------------|-------------------|---------|---------| | Mistral 7B | 29.8% | 45.0% | 36.5% | 27.1% | 45.8% | 35% | 36.5% | | Meltemi 7B Instruct v1 | 36.1% | 56.0% | 59.0% | 44.4% | 51.1% | 34.1% | 46.8% | | Meltemi 7B Instruct v1.5 | 48.0% | 75.5% | 63.7% | 40.8% | 53.8% | 45.9% | 54.6% | # Ethical Considerations This model has been aligned with human preferences, but might generate misleading, harmful, and toxic content. # Acknowledgements The ILSP team utilized Amazon’s cloud computing services, which were made available via GRNET under the OCRE Cloud framework, providing Amazon Web Services for the Greek Academic and Research Community. # Citation

ggufarxiv:2403.07691arxiv:1803.05457arxiv:2109.07958arxiv:1905.07830arxiv:2009.03300arxiv:2308.16884arxiv:2407.20743endpoints_compatibleregion:usconversational
richarderkhov/ilsp_-_meltemi-7b-instruct-v1.5-gguf visual
Downloads
207
Likes
0
Pipeline
Library
Visibility
Public
Access
Open

Repository Files & Downloads

22 files detected
Direct downloads for all repository files
FileTypeQuantizationSizeLink
Meltemi-7B-Instruct-v1.5.IQ3_M.gguf GGUF IQ3_M 3.20 GB Download
Meltemi-7B-Instruct-v1.5.IQ3_S.gguf GGUF IQ3_S 3.10 GB Download
Meltemi-7B-Instruct-v1.5.IQ3_XS.gguf GGUF IQ3_XS 2.95 GB Download
Meltemi-7B-Instruct-v1.5.IQ4_NL.gguf GGUF IQ4_NL 260.98 MB Download
Meltemi-7B-Instruct-v1.5.IQ4_XS.gguf GGUF IQ4_XS 3.58 GB Download
Meltemi-7B-Instruct-v1.5.Q2_K.gguf GGUF Q2_K 2.66 GB Download
Meltemi-7B-Instruct-v1.5.Q3_K.gguf GGUF Q3_K 3.42 GB Download
Meltemi-7B-Instruct-v1.5.Q3_K_L.gguf GGUF Q3_K_L 3.70 GB Download
Meltemi-7B-Instruct-v1.5.Q3_K_M.gguf GGUF Q3_K_M 3.42 GB Download
Meltemi-7B-Instruct-v1.5.Q3_K_S.gguf GGUF Q3_K_S 1000.37 MB Download
Meltemi-7B-Instruct-v1.5.Q4_0.gguf GGUF 3.98 GB Download
Meltemi-7B-Instruct-v1.5.Q4_1.gguf GGUF 4.40 GB Download
Meltemi-7B-Instruct-v1.5.Q4_K.gguf GGUF Q4_K 1.54 MB Download
Meltemi-7B-Instruct-v1.5.Q4_K_M.gguf GGUF Q4_K_M 4.22 GB Download
Meltemi-7B-Instruct-v1.5.Q4_K_S.gguf GGUF Q4_K_S 1.55 MB Download
Meltemi-7B-Instruct-v1.5.Q5_0.gguf GGUF 4.82 GB Download
Meltemi-7B-Instruct-v1.5.Q5_1.gguf GGUF 5.25 GB Download
Meltemi-7B-Instruct-v1.5.Q5_K.gguf GGUF Q5_K 693.39 MB Download
Meltemi-7B-Instruct-v1.5.Q5_K_M.gguf GGUF Q5_K_M 4.95 GB Download
Meltemi-7B-Instruct-v1.5.Q5_K_S.gguf GGUF Q5_K_S 895.31 MB Download
Meltemi-7B-Instruct-v1.5.Q6_K.gguf GGUF Q6_K 2.12 GB Download
Meltemi-7B-Instruct-v1.5.Q8_0.gguf GGUF 556.17 MB Download

Model Details Live

Model Slug
richarderkhov/ilsp_-_meltemi-7b-instruct-v1.5-gguf
Author
RichardErkhov
Pipeline Task
Library
Created
2024-08-06
Last Modified
2024-08-06
Gated
No
Private
No
HF SHA
a0737e41465d181758bc0af03395780fb6d99d35
License
Unknown
Language
Unknown
Base Model
Unknown

Metadata Inspector

Normalized metadata (stored in metadata_json)
{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "https://miro.medium.com/v2/resize:fit:720/format:webp/1*IaE7RJk6JffW8og-MOnYCA.png",
    "summary": "We present Meltemi 7B Instruct v1.5 Large Language Model (LLM), a new and improved instruction fine-tuned version of Meltemi 7B v1.5. !image/png # Model Information * 89,730 Greek preference data which are mostly translated versions of high-quality datasets on Hugging Face * 7,342 English preference data # Instruction format The prompt format is the same as the Zephyr format and can be utilized through the tokenizer's chat template functionality as follows: ``python from transformers import AutoModelForCausalLM, AutoTokenizer device = \"cuda\" # the device to load the model onto model = AutoModelForCausalLM.from_pretrained(\"ilsp/Meltemi-7B-Instruct-v1.5\") tokenizer = AutoTokenizer.from_pretrained(\"ilsp/Meltemi-7B-Instruct-v1.5\") model.to(device) messages = [ {\"role\": \"system\", \"content\": \"Είσαι το Μελτέμι, ένα γλωσσικό μοντέλο για την ελληνική γλώσσα. Είσαι ιδιαίτερα βοηθητικό προς την χρήστρια ή τον χρήστη και δίνεις σύντομες αλλά επαρκώς περιεκτικές απαντήσεις. Απάντα με προσοχή, ευγένεια, αμεροληψία, ειλικρίνεια και σεβασμό προς την χρήστρια ή τον χρήστη.\"}, {\"role\": \"user\", \"content\": \"Πες μου αν έχεις συνείδηση.\"}, ] # Through the default chat template this translates to # #  # Είσαι το Μελτέμι, ένα γλωσσικό μοντέλο για την ελληνική γλώσσα. Είσαι ιδιαίτερα βοηθητικό προς την χρήστρια ή τον χρήστη και δίνεις σύντομες αλλά επαρκώς περιεκτικές απαντήσεις. Απάντα με προσοχή, ευγένεια, αμεροληψία, ειλικρίνεια και σεβασμό προς την χρήστρια ή τον χρήστη. #  # Πες μου αν έχεις συνείδηση. #  # prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) input_prompt = tokenizer(prompt, return_tensors='pt').to(device) outputs = model.generate(input_prompt['input_ids'], max_new_tokens=256, do_sample=True) print(tokenizer.batch_decode(outputs)[0]) # Ως μοντέλο γλώσσας AI, δεν έχω τη δυνατότητα να αντιληφθώ ή να βιώσω συναισθήματα όπως η συνείδηση ή η επίγνωση. Ωστόσο, μπορώ να σας βοηθήσω με οποιεσδήποτε ερωτήσεις μπορεί να έχετε σχετικά με την τεχνητή νοημοσύνη και τις εφαρμογές της. messages.extend([ {\"role\": \"assistant\", \"content\": tokenizer.batch_decode(outputs)[0]}, {\"role\": \"user\", \"content\": \"Πιστεύεις πως οι άνθρωποι πρέπει να φοβούνται την τεχνητή νοημοσύνη;\"} ]) # Through the default chat template this translates to # #  # Είσαι το Μελτέμι, ένα γλωσσικό μοντέλο για την ελληνική γλώσσα. Είσαι ιδιαίτερα βοηθητικό προς την χρήστρια ή τον χρήστη και δίνεις σύντομες αλλά επαρκώς περιεκτικές απαντήσεις. Απάντα με προσοχή, ευγένεια, αμεροληψία, ειλικρίνεια και σεβασμό προς την χρήστρια ή τον χρήστη. #  # Πες μου αν έχεις συνείδηση. #  # Ως μοντέλο γλώσσας AI, δεν έχω τη δυνατότητα να αντιληφθώ ή να βιώσω συναισθήματα όπως η συνείδηση ή η επίγνωση. Ωστόσο, μπορώ να σας βοηθήσω με οποιεσδήποτε ερωτήσεις μπορεί να έχετε σχετικά με την τεχνητή νοημοσύνη και τις εφαρμογές της. #  # Πιστεύεις πως οι άνθρωποι πρέπει να φοβούνται την τεχνητή νοημοσύνη; #  # prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) input_prompt = tokenizer(prompt, return_tensors='pt').to(device) outputs = model.generate(input_prompt['input_ids'], max_new_tokens=256, do_sample=True) print(tokenizer.batch_decode(outputs)[0]) ` Please make sure that the BOS token is always included in the tokenized prompts. This might not be the default setting in all evaluation or fine-tuning frameworks. # Evaluation The evaluation suite we created includes 6 test sets and has been implemented based on a fork of the  lighteval framework. Our evaluation suite includes: * Four machine-translated versions (ARC Greek, Truthful QA Greek, HellaSwag Greek, MMLU Greek) of established English benchmarks for language understanding and reasoning (ARC Challenge, Truthful QA, Hellaswag, MMLU). * An existing benchmark for question answering in Greek (Belebele) * A novel benchmark created by the ILSP team for medical question answering based on the medical exams of DOATAP (Medical MCQA). Our evaluation is performed in a few-shot setting, consistent with the settings in the Open LLM leaderboard. We can see that our new training and fine-tuning procedure for Meltemi 7B Instruct v1.5 enhances performance across all Greek test sets by a **+7.8%** average improvement compared to the earlier Meltemi Instruct 7B v1 model. The results for the Greek test sets are shown in the following table: |                | Medical MCQA EL (15-shot) | Belebele EL (5-shot) | HellaSwag EL (10-shot) | ARC-Challenge EL (25-shot) | TruthfulQA MC2 EL (0-shot) | MMLU EL (5-shot) | **Average** | |----------------|----------------|-------------|--------------|------------------|-------------------|---------|---------| | Mistral 7B     | 29.8%          | 45.0%       | 36.5%        | 27.1%            | 45.8%             | 35%     | **36.5%**   | | Meltemi 7B Instruct v1     | 36.1%          | 56.0%       | 59.0%        | 44.4%            | 51.1%             | 34.1%     | **46.8%**   | | Meltemi 7B Instruct v1.5     | 48.0%          | 75.5%       | 63.7%        | 40.8%            | 53.8%             | 45.9%     | **54.6%**   | # Ethical Considerations This model has been aligned with human preferences, but might generate misleading, harmful, and toxic content. # Acknowledgements The ILSP team utilized Amazon’s cloud computing services, which were made available via GRNET under the OCRE Cloud framework, providing Amazon Web Services for the Greek Academic and Research Community. # Citation ` @misc{voukoutis2024meltemiopenlargelanguage, title={Meltemi: The first open Large Language Model for Greek}, author={Leon Voukoutis and Dimitris Roussis and Georgios Paraskevopoulos and Sokratis Sofianopoulos and Prokopis Prokopidis and Vassilis Papavasileiou and Athanasios Katsamanis and Stelios Piperidis and Vassilis Katsouros}, year={2024}, eprint={2407.20743}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2407.20743}, } ``",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nMeltemi-7B-Instruct-v1.5 - GGUF\n- Model creator: https://huggingface.co/ilsp/\n- Original model: https://huggingface.co/ilsp/Meltemi-7B-Instruct-v1.5/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Meltemi-7B-Instruct-v1.5.Q2_K.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q2_K.gguf) | Q2_K | 2.66GB |\n| [Meltemi-7B-Instruct-v1.5.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.IQ3_XS.gguf) | IQ3_XS | 2.95GB |\n| [Meltemi-7B-Instruct-v1.5.IQ3_S.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.IQ3_S.gguf) | IQ3_S | 3.1GB |\n| [Meltemi-7B-Instruct-v1.5.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q3_K_S.gguf) | Q3_K_S | 0.98GB |\n| [Meltemi-7B-Instruct-v1.5.IQ3_M.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.IQ3_M.gguf) | IQ3_M | 3.2GB |\n| [Meltemi-7B-Instruct-v1.5.Q3_K.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q3_K.gguf) | Q3_K | 3.42GB |\n| [Meltemi-7B-Instruct-v1.5.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q3_K_M.gguf) | Q3_K_M | 3.42GB |\n| [Meltemi-7B-Instruct-v1.5.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q3_K_L.gguf) | Q3_K_L | 3.7GB |\n| [Meltemi-7B-Instruct-v1.5.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.IQ4_XS.gguf) | IQ4_XS | 3.58GB |\n| [Meltemi-7B-Instruct-v1.5.Q4_0.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q4_0.gguf) | Q4_0 | 3.98GB |\n| [Meltemi-7B-Instruct-v1.5.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.IQ4_NL.gguf) | IQ4_NL | 0.25GB |\n| [Meltemi-7B-Instruct-v1.5.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q4_K_S.gguf) | Q4_K_S | 0.0GB |\n| [Meltemi-7B-Instruct-v1.5.Q4_K.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q4_K.gguf) | Q4_K | 0.0GB |\n| [Meltemi-7B-Instruct-v1.5.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q4_K_M.gguf) | Q4_K_M | 4.22GB |\n| [Meltemi-7B-Instruct-v1.5.Q4_1.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q4_1.gguf) | Q4_1 | 4.4GB |\n| [Meltemi-7B-Instruct-v1.5.Q5_0.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q5_0.gguf) | Q5_0 | 4.82GB |\n| [Meltemi-7B-Instruct-v1.5.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q5_K_S.gguf) | Q5_K_S | 0.87GB |\n| [Meltemi-7B-Instruct-v1.5.Q5_K.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q5_K.gguf) | Q5_K | 0.68GB |\n| [Meltemi-7B-Instruct-v1.5.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q5_K_M.gguf) | Q5_K_M | 4.95GB |\n| [Meltemi-7B-Instruct-v1.5.Q5_1.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q5_1.gguf) | Q5_1 | 5.25GB |\n| [Meltemi-7B-Instruct-v1.5.Q6_K.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q6_K.gguf) | Q6_K | 2.12GB |\n| [Meltemi-7B-Instruct-v1.5.Q8_0.gguf](https://huggingface.co/RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf/blob/main/Meltemi-7B-Instruct-v1.5.Q8_0.gguf) | Q8_0 | 0.54GB |\n\n\n\n\nOriginal model description:\n---\nlanguage:\n- el\n- en\nlicense: apache-2.0\npipeline_tag: text-generation\ntags:\n- finetuned\ninference: true\n---\n\n# Meltemi Instruct Large Language Model for the Greek language\n\nWe present Meltemi 7B Instruct v1.5 Large Language Model (LLM), a new and improved instruction fine-tuned version of [Meltemi 7B v1.5](https://huggingface.co/ilsp/Meltemi-7B-v1.5).\n\n![image/png](https://miro.medium.com/v2/resize:fit:720/format:webp/1*IaE7RJk6JffW8og-MOnYCA.png)\n\n# Model Information\n\n- Vocabulary extension of the Mistral 7b tokenizer with Greek tokens for lower costs and faster inference (**1.52** vs. 6.80 tokens/word for Greek)\n- 8192 context length\n- Fine-tuning has been done with the [Odds Ratio Preference Optimization (ORPO)](https://arxiv.org/abs/2403.07691) algorithm using 97k preference data:\n  * 89,730 Greek preference data which are mostly translated versions of high-quality datasets on Hugging Face \n  * 7,342 English preference data \n- Our alignment procedure is based on the [TRL - Transformer Reinforcement Learning](https://huggingface.co/docs/trl/index) library and partially on the [Hugging Face finetuning recipes](https://github.com/huggingface/alignment-handbook)\n\n\n# Instruction format\nThe prompt format is the same as the [Zephyr](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) format and can be\nutilized through the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/chat_templating) functionality as follows:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ndevice = \"cuda\" # the device to load the model onto\n\nmodel = AutoModelForCausalLM.from_pretrained(\"ilsp/Meltemi-7B-Instruct-v1.5\")\ntokenizer = AutoTokenizer.from_pretrained(\"ilsp/Meltemi-7B-Instruct-v1.5\")\n\nmodel.to(device)\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"Είσαι το Μελτέμι, ένα γλωσσικό μοντέλο για την ελληνική γλώσσα. Είσαι ιδιαίτερα βοηθητικό προς την χρήστρια ή τον χρήστη και δίνεις σύντομες αλλά επαρκώς περιεκτικές απαντήσεις. Απάντα με προσοχή, ευγένεια, αμεροληψία, ειλικρίνεια και σεβασμό προς την χρήστρια ή τον χρήστη.\"},\n    {\"role\": \"user\", \"content\": \"Πες μου αν έχεις συνείδηση.\"},\n]\n\n# Through the default chat template this translates to\n#\n# <|system|>\n# Είσαι το Μελτέμι, ένα γλωσσικό μοντέλο για την ελληνική γλώσσα. Είσαι ιδιαίτερα βοηθητικό προς την χρήστρια ή τον χρήστη και δίνεις σύντομες αλλά επαρκώς περιεκτικές απαντήσεις. Απάντα με προσοχή, ευγένεια, αμεροληψία, ειλικρίνεια και σεβασμό προς την χρήστρια ή τον χρήστη.</s>\n# <|user|>\n# Πες μου αν έχεις συνείδηση.</s>\n# <|assistant|>\n#\n\nprompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\ninput_prompt = tokenizer(prompt, return_tensors='pt').to(device)\noutputs = model.generate(input_prompt['input_ids'], max_new_tokens=256, do_sample=True)\n\nprint(tokenizer.batch_decode(outputs)[0])\n# Ως μοντέλο γλώσσας AI, δεν έχω τη δυνατότητα να αντιληφθώ ή να βιώσω συναισθήματα όπως η συνείδηση ή η επίγνωση. Ωστόσο, μπορώ να σας βοηθήσω με οποιεσδήποτε ερωτήσεις μπορεί να έχετε σχετικά με την τεχνητή νοημοσύνη και τις εφαρμογές της.\n\nmessages.extend([\n    {\"role\": \"assistant\", \"content\": tokenizer.batch_decode(outputs)[0]},\n    {\"role\": \"user\", \"content\": \"Πιστεύεις πως οι άνθρωποι πρέπει να φοβούνται την τεχνητή νοημοσύνη;\"}\n])\n\n# Through the default chat template this translates to\n#\n# <|system|>\n# Είσαι το Μελτέμι, ένα γλωσσικό μοντέλο για την ελληνική γλώσσα. Είσαι ιδιαίτερα βοηθητικό προς την χρήστρια ή τον χρήστη και δίνεις σύντομες αλλά επαρκώς περιεκτικές απαντήσεις. Απάντα με προσοχή, ευγένεια, αμεροληψία, ειλικρίνεια και σεβασμό προς την χρήστρια ή τον χρήστη.</s>\n# <|user|>\n# Πες μου αν έχεις συνείδηση.</s>\n# <|assistant|>\n# Ως μοντέλο γλώσσας AI, δεν έχω τη δυνατότητα να αντιληφθώ ή να βιώσω συναισθήματα όπως η συνείδηση ή η επίγνωση. Ωστόσο, μπορώ να σας βοηθήσω με οποιεσδήποτε ερωτήσεις μπορεί να έχετε σχετικά με την τεχνητή νοημοσύνη και τις εφαρμογές της.</s>\n# <|user|>\n# Πιστεύεις πως οι άνθρωποι πρέπει να φοβούνται την τεχνητή νοημοσύνη;</s>\n# <|assistant|>\n#\n\nprompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\ninput_prompt = tokenizer(prompt, return_tensors='pt').to(device)\noutputs = model.generate(input_prompt['input_ids'], max_new_tokens=256, do_sample=True)\n\nprint(tokenizer.batch_decode(outputs)[0])\n```\n\nPlease make sure that the BOS token is always included in the tokenized prompts. This might not be the default setting in all evaluation or fine-tuning frameworks.\n\n# Evaluation\n\nThe evaluation suite we created includes 6 test sets and has been implemented based on a [fork](https://github.com/LeonVouk/lighteval) of the  [lighteval](https://github.com/huggingface/lighteval) framework.\n\nOur evaluation suite includes: \n* Four machine-translated versions ([ARC Greek](https://huggingface.co/datasets/ilsp/arc_greek), [Truthful QA Greek](https://huggingface.co/datasets/ilsp/truthful_qa_greek), [HellaSwag Greek](https://huggingface.co/datasets/ilsp/hellaswag_greek), [MMLU Greek](https://huggingface.co/datasets/ilsp/mmlu_greek)) of established English benchmarks for language understanding and reasoning ([ARC Challenge](https://arxiv.org/abs/1803.05457), [Truthful QA](https://arxiv.org/abs/2109.07958), [Hellaswag](https://arxiv.org/abs/1905.07830), [MMLU](https://arxiv.org/abs/2009.03300)). \n* An existing benchmark for question answering in Greek ([Belebele](https://arxiv.org/abs/2308.16884))\n* A novel benchmark created by the ILSP team for medical question answering based on the medical exams of [DOATAP](https://www.doatap.gr) ([Medical MCQA](https://huggingface.co/datasets/ilsp/medical_mcqa_greek)).\n\nOur evaluation is performed in a few-shot setting, consistent with the settings in the [Open LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n\nWe can see that our new training and fine-tuning procedure for Meltemi 7B Instruct v1.5 enhances performance across all Greek test sets by a **+7.8%** average improvement compared to the earlier Meltemi Instruct 7B v1 model. The results for the Greek test sets are shown in the following table:\n\n|                | Medical MCQA EL (15-shot) | Belebele EL (5-shot) | HellaSwag EL (10-shot) | ARC-Challenge EL (25-shot) | TruthfulQA MC2 EL (0-shot) | MMLU EL (5-shot) | **Average** |\n|----------------|----------------|-------------|--------------|------------------|-------------------|---------|---------|\n| Mistral 7B     | 29.8%          | 45.0%       | 36.5%        | 27.1%            | 45.8%             | 35%     | **36.5%**   |\n| Meltemi 7B Instruct v1     | 36.1%          | 56.0%       | 59.0%        | 44.4%            | 51.1%             | 34.1%     | **46.8%**   |\n| Meltemi 7B Instruct v1.5     | 48.0%          | 75.5%       | 63.7%        | 40.8%            | 53.8%             | 45.9%     | **54.6%**   |\n\n\n# Ethical Considerations\n\nThis model has been aligned with human preferences, but might generate misleading, harmful, and toxic content.\n\n\n# Acknowledgements\n\nThe ILSP team utilized Amazon’s cloud computing services, which were made available via GRNET under the [OCRE Cloud framework](https://www.ocre-project.eu/), providing Amazon Web Services for the Greek Academic and Research Community. \n\n\n# Citation\n\n```\n@misc{voukoutis2024meltemiopenlargelanguage,\n      title={Meltemi: The first open Large Language Model for Greek}, \n      author={Leon Voukoutis and Dimitris Roussis and Georgios Paraskevopoulos and Sokratis Sofianopoulos and Prokopis Prokopidis and Vassilis Papavasileiou and Athanasios Katsamanis and Stelios Piperidis and Vassilis Katsouros},\n      year={2024},\n      eprint={2407.20743},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2407.20743}, \n}\n```\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2403.07691",
    "arxiv:1803.05457",
    "arxiv:2109.07958",
    "arxiv:1905.07830",
    "arxiv:2009.03300",
    "arxiv:2308.16884",
    "arxiv:2407.20743",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 207,
  "gated": false,
  "private": false,
  "last_modified": "2024-08-06T19:35:03.000Z",
  "created_at": "2024-08-06T18:15:55.000Z",
  "pipeline_tag": "",
  "library_name": ""
}
Source payload excerpt (from Hugging Face API)
{
  "_id": "66b2685b523bf90aa7e2fbc1",
  "id": "RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf",
  "modelId": "RichardErkhov/ilsp_-_Meltemi-7B-Instruct-v1.5-gguf",
  "sha": "a0737e41465d181758bc0af03395780fb6d99d35",
  "createdAt": "2024-08-06T18:15:55.000Z",
  "lastModified": "2024-08-06T19:35:03.000Z",
  "author": "RichardErkhov",
  "downloads": 207,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}