Model Intelligence Sheet
richarderkhov/alibaba-nlp_-_gte-qwen2-1.5b-instruct-gguf overview
model.maxseqlength = 8192 queries = [ "how much protein should a female eat", "summit define", ] documents = [ "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.", "Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.", ] queryembeddings = model.encode(queries, promptname="query") documentembeddings = model.encode(documents) scores = (queryembeddings @ documentembeddings.T) * 100 print(scores.tolist()) python import torch import torch.nn.functional as F from torch import Tensor from transformers import AutoTokenizer, AutoModel def lasttokenpool(lasthiddenstates: Tensor, attentionmask: Tensor) -> Tensor: leftpadding = (attentionmask[:, -1].sum() == attentionmask.shape[0]) if leftpadding: return lasthiddenstates[:, -1] else: sequencelengths = attentionmask.sum(dim=1) - 1 batchsize = lasthiddenstates.shape[0] return lasthiddenstates[torch.arange(batchsize, device=lasthiddenstates.device), sequencelengths] def getdetailedinstruct(taskdescription: str, query: str) -> str: return f'Instruct: {taskdescription}\nQuery: {query}' # Each query must come with a one-sentence instruction that describes the task task = 'Given a web search query, retrieve relevant passages that answer the query' queries = [ getdetailedinstruct(task, 'how much protein should a female eat'), getdetailedinstruct(task, 'summit define') ] # No need to add instruction for retrieval documents documents = [ "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.", "Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments." ] inputtexts = queries + documents tokenizer = AutoTokenizer.frompretrained('Alibaba-NLP/gte-Qwen2-1.5B-instruct', trustremotecode=True) model = AutoModel.frompretrained('Alibaba-NLP/gte-Qwen2-1.5B-instruct', trustremotecode=True) maxlength = 8192 # Tokenize the input texts batchdict = tokenizer(inputtexts, maxlength=maxlength, padding=True, truncation=True, returntensors='pt') outputs = model(batchdict) embeddings = lasttokenpool(outputs.lasthiddenstate, batchdict['attention_mask']) # normalize embeddings embeddings = F.normalize(embeddings, p=2, dim=1) scores = (embeddings[:2] @ embeddings[2:].T) * 100 print(scores.tolist())
Downloads
112
Likes
0
Pipeline
—
Library
—
Visibility
Public
Access
Open
Repository Files & Downloads
22 files detected
Direct downloads for all repository files
| File | Type | Quantization | Size | Link |
|---|---|---|---|---|
| gte-Qwen2-1.5B-instruct.IQ3_M.gguf | GGUF | IQ3_M | 835.78 MB | Download |
| gte-Qwen2-1.5B-instruct.IQ3_S.gguf | GGUF | IQ3_S | 822.18 MB | Download |
| gte-Qwen2-1.5B-instruct.IQ3_XS.gguf | GGUF | IQ3_XS | 792.89 MB | Download |
| gte-Qwen2-1.5B-instruct.IQ4_NL.gguf | GGUF | IQ4_NL | 1022.47 MB | Download |
| gte-Qwen2-1.5B-instruct.IQ4_XS.gguf | GGUF | IQ4_XS | 978.04 MB | Download |
| gte-Qwen2-1.5B-instruct.Q2_K.gguf | GGUF | Q2_K | 717.51 MB | Download |
| gte-Qwen2-1.5B-instruct.Q3_K.gguf | GGUF | Q3_K | 881.09 MB | Download |
| gte-Qwen2-1.5B-instruct.Q3_K_L.gguf | GGUF | Q3_K_L | 934.48 MB | Download |
| gte-Qwen2-1.5B-instruct.Q3_K_M.gguf | GGUF | Q3_K_M | 881.09 MB | Download |
| gte-Qwen2-1.5B-instruct.Q3_K_S.gguf | GGUF | Q3_K_S | 820.79 MB | Download |
| gte-Qwen2-1.5B-instruct.Q4_0.gguf | GGUF | — | 1016.24 MB | Download |
| gte-Qwen2-1.5B-instruct.Q4_1.gguf | GGUF | — | 1.08 GB | Download |
| gte-Qwen2-1.5B-instruct.Q4_K.gguf | GGUF | Q4_K | 1.04 GB | Download |
| gte-Qwen2-1.5B-instruct.Q4_K_M.gguf | GGUF | Q4_K_M | 1.04 GB | Download |
| gte-Qwen2-1.5B-instruct.Q4_K_S.gguf | GGUF | Q4_K_S | 1021.35 MB | Download |
| gte-Qwen2-1.5B-instruct.Q5_0.gguf | GGUF | — | 1.17 GB | Download |
| gte-Qwen2-1.5B-instruct.Q5_1.gguf | GGUF | — | 1.26 GB | Download |
| gte-Qwen2-1.5B-instruct.Q5_K.gguf | GGUF | Q5_K | 1.20 GB | Download |
| gte-Qwen2-1.5B-instruct.Q5_K_M.gguf | GGUF | Q5_K_M | 1.20 GB | Download |
| gte-Qwen2-1.5B-instruct.Q5_K_S.gguf | GGUF | Q5_K_S | 1.17 GB | Download |
| gte-Qwen2-1.5B-instruct.Q6_K.gguf | GGUF | Q6_K | 1.36 GB | Download |
| gte-Qwen2-1.5B-instruct.Q8_0.gguf | GGUF | — | 1.76 GB | Download |
Model Details Live
Metadata Inspector
Normalized metadata (stored in metadata_json)
{
"metadata": {},
"card_data": {
"frontmatter": {},
"hero_image_url": "",
"summary": "model.max_seq_length = 8192 queries = [ \"how much protein should a female eat\", \"summit define\", ] documents = [ \"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.\", \"Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.\", ] query_embeddings = model.encode(queries, prompt_name=\"query\") document_embeddings = model.encode(documents) scores = (query_embeddings @ document_embeddings.T) * 100 print(scores.tolist()) `` Observe the config_sentence_transformers.json to see all pre-built prompt names. Otherwise, you can use model.encode(queries, prompt=\"Instruct: ...\\nQuery: \" to use a custom prompt of your choice. ### Transformers `python import torch import torch.nn.functional as F from torch import Tensor from transformers import AutoTokenizer, AutoModel def last_token_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor: left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0]) if left_padding: return last_hidden_states[:, -1] else: sequence_lengths = attention_mask.sum(dim=1) - 1 batch_size = last_hidden_states.shape[0] return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths] def get_detailed_instruct(task_description: str, query: str) -> str: return f'Instruct: {task_description}\\nQuery: {query}' # Each query must come with a one-sentence instruction that describes the task task = 'Given a web search query, retrieve relevant passages that answer the query' queries = [ get_detailed_instruct(task, 'how much protein should a female eat'), get_detailed_instruct(task, 'summit define') ] # No need to add instruction for retrieval documents documents = [ \"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.\", \"Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.\" ] input_texts = queries + documents tokenizer = AutoTokenizer.from_pretrained('Alibaba-NLP/gte-Qwen2-1.5B-instruct', trust_remote_code=True) model = AutoModel.from_pretrained('Alibaba-NLP/gte-Qwen2-1.5B-instruct', trust_remote_code=True) max_length = 8192 # Tokenize the input texts batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors='pt') outputs = model(**batch_dict) embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask']) # normalize embeddings embeddings = F.normalize(embeddings, p=2, dim=1) scores = (embeddings[:2] @ embeddings[2:].T) * 100 print(scores.tolist()) ``",
"quick_links": [],
"benchmark_table_html": "",
"readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\ngte-Qwen2-1.5B-instruct - GGUF\n- Model creator: https://huggingface.co/Alibaba-NLP/\n- Original model: https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [gte-Qwen2-1.5B-instruct.Q2_K.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q2_K.gguf) | Q2_K | 0.7GB |\n| [gte-Qwen2-1.5B-instruct.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.IQ3_XS.gguf) | IQ3_XS | 0.77GB |\n| [gte-Qwen2-1.5B-instruct.IQ3_S.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.IQ3_S.gguf) | IQ3_S | 0.8GB |\n| [gte-Qwen2-1.5B-instruct.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q3_K_S.gguf) | Q3_K_S | 0.8GB |\n| [gte-Qwen2-1.5B-instruct.IQ3_M.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.IQ3_M.gguf) | IQ3_M | 0.82GB |\n| [gte-Qwen2-1.5B-instruct.Q3_K.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q3_K.gguf) | Q3_K | 0.86GB |\n| [gte-Qwen2-1.5B-instruct.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q3_K_M.gguf) | Q3_K_M | 0.86GB |\n| [gte-Qwen2-1.5B-instruct.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q3_K_L.gguf) | Q3_K_L | 0.91GB |\n| [gte-Qwen2-1.5B-instruct.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.IQ4_XS.gguf) | IQ4_XS | 0.96GB |\n| [gte-Qwen2-1.5B-instruct.Q4_0.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q4_0.gguf) | Q4_0 | 0.99GB |\n| [gte-Qwen2-1.5B-instruct.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.IQ4_NL.gguf) | IQ4_NL | 1.0GB |\n| [gte-Qwen2-1.5B-instruct.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q4_K_S.gguf) | Q4_K_S | 1.0GB |\n| [gte-Qwen2-1.5B-instruct.Q4_K.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q4_K.gguf) | Q4_K | 1.04GB |\n| [gte-Qwen2-1.5B-instruct.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q4_K_M.gguf) | Q4_K_M | 1.04GB |\n| [gte-Qwen2-1.5B-instruct.Q4_1.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q4_1.gguf) | Q4_1 | 1.08GB |\n| [gte-Qwen2-1.5B-instruct.Q5_0.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q5_0.gguf) | Q5_0 | 1.17GB |\n| [gte-Qwen2-1.5B-instruct.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q5_K_S.gguf) | Q5_K_S | 1.17GB |\n| [gte-Qwen2-1.5B-instruct.Q5_K.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q5_K.gguf) | Q5_K | 1.2GB |\n| [gte-Qwen2-1.5B-instruct.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q5_K_M.gguf) | Q5_K_M | 1.2GB |\n| [gte-Qwen2-1.5B-instruct.Q5_1.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q5_1.gguf) | Q5_1 | 1.26GB |\n| [gte-Qwen2-1.5B-instruct.Q6_K.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q6_K.gguf) | Q6_K | 1.36GB |\n| [gte-Qwen2-1.5B-instruct.Q8_0.gguf](https://huggingface.co/RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf/blob/main/gte-Qwen2-1.5B-instruct.Q8_0.gguf) | Q8_0 | 1.76GB |\n\n\n\n\nOriginal model description:\n---\ntags:\n- mteb\n- sentence-transformers\n- transformers\n- Qwen2\n- sentence-similarity\nlicense: apache-2.0\nmodel-index:\n- name: gte-qwen2-7B-instruct\n results:\n - dataset:\n config: en\n name: MTEB AmazonCounterfactualClassification (en)\n revision: e8379541af4e31359cca9fbcf4b00f2671dba205\n split: test\n type: mteb/amazon_counterfactual\n metrics:\n - type: accuracy\n value: 83.98507462686567\n - type: ap\n value: 50.93015252587014\n - type: f1\n value: 78.50416599051215\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB AmazonPolarityClassification\n revision: e2d317d38cd51312af73b3d32a06d1a08b442046\n split: test\n type: mteb/amazon_polarity\n metrics:\n - type: accuracy\n value: 96.61065\n - type: ap\n value: 94.89174052954196\n - type: f1\n value: 96.60942596940565\n task:\n type: Classification\n - dataset:\n config: en\n name: MTEB AmazonReviewsClassification (en)\n revision: 1399c76144fd37290681b995c656ef9b2e06e26d\n split: test\n type: mteb/amazon_reviews_multi\n metrics:\n - type: accuracy\n value: 55.614000000000004\n - type: f1\n value: 54.90553480294904\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB ArguAna\n revision: c22ab2a51041ffd869aaddef7af8d8215647e41a\n split: test\n type: mteb/arguana\n metrics:\n - type: map_at_1\n value: 45.164\n - type: map_at_10\n value: 61.519\n - type: map_at_100\n value: 61.769\n - type: map_at_1000\n value: 61.769\n - type: map_at_3\n value: 57.443999999999996\n - type: map_at_5\n value: 60.058\n - type: mrr_at_1\n value: 46.088\n - type: mrr_at_10\n value: 61.861\n - type: mrr_at_100\n value: 62.117999999999995\n - type: mrr_at_1000\n value: 62.117999999999995\n - type: mrr_at_3\n value: 57.729\n - type: mrr_at_5\n value: 60.392\n - type: ndcg_at_1\n value: 45.164\n - type: ndcg_at_10\n value: 69.72\n - type: ndcg_at_100\n value: 70.719\n - type: ndcg_at_1000\n value: 70.719\n - type: ndcg_at_3\n value: 61.517999999999994\n - type: ndcg_at_5\n value: 66.247\n - type: precision_at_1\n value: 45.164\n - type: precision_at_10\n value: 9.545\n - type: precision_at_100\n value: 0.996\n - type: precision_at_1000\n value: 0.1\n - type: precision_at_3\n value: 24.443\n - type: precision_at_5\n value: 16.97\n - type: recall_at_1\n value: 45.164\n - type: recall_at_10\n value: 95.448\n - type: recall_at_100\n value: 99.644\n - type: recall_at_1000\n value: 99.644\n - type: recall_at_3\n value: 73.329\n - type: recall_at_5\n value: 84.851\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB ArxivClusteringP2P\n revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d\n split: test\n type: mteb/arxiv-clustering-p2p\n metrics:\n - type: v_measure\n value: 50.511868162026175\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB ArxivClusteringS2S\n revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53\n split: test\n type: mteb/arxiv-clustering-s2s\n metrics:\n - type: v_measure\n value: 45.007803189284004\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB AskUbuntuDupQuestions\n revision: 2000358ca161889fa9c082cb41daa8dcfb161a54\n split: test\n type: mteb/askubuntudupquestions-reranking\n metrics:\n - type: map\n value: 64.55292107723382\n - type: mrr\n value: 77.66158818097877\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB BIOSSES\n revision: d3fb88f8f02e40887cd149695127462bbcf29b4a\n split: test\n type: mteb/biosses-sts\n metrics:\n - type: cos_sim_pearson\n value: 85.65459047085452\n - type: cos_sim_spearman\n value: 82.10729255710761\n - type: euclidean_pearson\n value: 82.78079159312476\n - type: euclidean_spearman\n value: 80.50002701880933\n - type: manhattan_pearson\n value: 82.41372641383016\n - type: manhattan_spearman\n value: 80.57412509272639\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB Banking77Classification\n revision: 0fd18e25b25c072e09e0d92ab615fda904d66300\n split: test\n type: mteb/banking77\n metrics:\n - type: accuracy\n value: 87.30844155844156\n - type: f1\n value: 87.25307322443255\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB BiorxivClusteringP2P\n revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40\n split: test\n type: mteb/biorxiv-clustering-p2p\n metrics:\n - type: v_measure\n value: 43.20754608934859\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB BiorxivClusteringS2S\n revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908\n split: test\n type: mteb/biorxiv-clustering-s2s\n metrics:\n - type: v_measure\n value: 38.818037697335505\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB CQADupstackAndroidRetrieval\n revision: f46a197baaae43b4f621051089b82a364682dfeb\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 35.423\n - type: map_at_10\n value: 47.198\n - type: map_at_100\n value: 48.899\n - type: map_at_1000\n value: 49.004\n - type: map_at_3\n value: 43.114999999999995\n - type: map_at_5\n value: 45.491\n - type: mrr_at_1\n value: 42.918\n - type: mrr_at_10\n value: 53.299\n - type: mrr_at_100\n value: 54.032000000000004\n - type: mrr_at_1000\n value: 54.055\n - type: mrr_at_3\n value: 50.453\n - type: mrr_at_5\n value: 52.205999999999996\n - type: ndcg_at_1\n value: 42.918\n - type: ndcg_at_10\n value: 53.98\n - type: ndcg_at_100\n value: 59.57\n - type: ndcg_at_1000\n value: 60.879000000000005\n - type: ndcg_at_3\n value: 48.224000000000004\n - type: ndcg_at_5\n value: 50.998\n - type: precision_at_1\n value: 42.918\n - type: precision_at_10\n value: 10.299999999999999\n - type: precision_at_100\n value: 1.687\n - type: precision_at_1000\n value: 0.211\n - type: precision_at_3\n value: 22.842000000000002\n - type: precision_at_5\n value: 16.681\n - type: recall_at_1\n value: 35.423\n - type: recall_at_10\n value: 66.824\n - type: recall_at_100\n value: 89.564\n - type: recall_at_1000\n value: 97.501\n - type: recall_at_3\n value: 50.365\n - type: recall_at_5\n value: 57.921\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackEnglishRetrieval\n revision: ad9991cb51e31e31e430383c75ffb2885547b5f0\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 33.205\n - type: map_at_10\n value: 44.859\n - type: map_at_100\n value: 46.135\n - type: map_at_1000\n value: 46.259\n - type: map_at_3\n value: 41.839\n - type: map_at_5\n value: 43.662\n - type: mrr_at_1\n value: 41.146\n - type: mrr_at_10\n value: 50.621\n - type: mrr_at_100\n value: 51.207\n - type: mrr_at_1000\n value: 51.246\n - type: mrr_at_3\n value: 48.535000000000004\n - type: mrr_at_5\n value: 49.818\n - type: ndcg_at_1\n value: 41.146\n - type: ndcg_at_10\n value: 50.683\n - type: ndcg_at_100\n value: 54.82\n - type: ndcg_at_1000\n value: 56.69\n - type: ndcg_at_3\n value: 46.611000000000004\n - type: ndcg_at_5\n value: 48.66\n - type: precision_at_1\n value: 41.146\n - type: precision_at_10\n value: 9.439\n - type: precision_at_100\n value: 1.465\n - type: precision_at_1000\n value: 0.194\n - type: precision_at_3\n value: 22.59\n - type: precision_at_5\n value: 15.86\n - type: recall_at_1\n value: 33.205\n - type: recall_at_10\n value: 61.028999999999996\n - type: recall_at_100\n value: 78.152\n - type: recall_at_1000\n value: 89.59700000000001\n - type: recall_at_3\n value: 49.05\n - type: recall_at_5\n value: 54.836\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackGamingRetrieval\n revision: 4885aa143210c98657558c04aaf3dc47cfb54340\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 41.637\n - type: map_at_10\n value: 55.162\n - type: map_at_100\n value: 56.142\n - type: map_at_1000\n value: 56.188\n - type: map_at_3\n value: 51.564\n - type: map_at_5\n value: 53.696\n - type: mrr_at_1\n value: 47.524\n - type: mrr_at_10\n value: 58.243\n - type: mrr_at_100\n value: 58.879999999999995\n - type: mrr_at_1000\n value: 58.9\n - type: mrr_at_3\n value: 55.69499999999999\n - type: mrr_at_5\n value: 57.284\n - type: ndcg_at_1\n value: 47.524\n - type: ndcg_at_10\n value: 61.305\n - type: ndcg_at_100\n value: 65.077\n - type: ndcg_at_1000\n value: 65.941\n - type: ndcg_at_3\n value: 55.422000000000004\n - type: ndcg_at_5\n value: 58.516\n - type: precision_at_1\n value: 47.524\n - type: precision_at_10\n value: 9.918000000000001\n - type: precision_at_100\n value: 1.276\n - type: precision_at_1000\n value: 0.13899999999999998\n - type: precision_at_3\n value: 24.765\n - type: precision_at_5\n value: 17.204\n - type: recall_at_1\n value: 41.637\n - type: recall_at_10\n value: 76.185\n - type: recall_at_100\n value: 92.149\n - type: recall_at_1000\n value: 98.199\n - type: recall_at_3\n value: 60.856\n - type: recall_at_5\n value: 68.25099999999999\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackGisRetrieval\n revision: 5003b3064772da1887988e05400cf3806fe491f2\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 26.27\n - type: map_at_10\n value: 37.463\n - type: map_at_100\n value: 38.434000000000005\n - type: map_at_1000\n value: 38.509\n - type: map_at_3\n value: 34.226\n - type: map_at_5\n value: 36.161\n - type: mrr_at_1\n value: 28.588\n - type: mrr_at_10\n value: 39.383\n - type: mrr_at_100\n value: 40.23\n - type: mrr_at_1000\n value: 40.281\n - type: mrr_at_3\n value: 36.422\n - type: mrr_at_5\n value: 38.252\n - type: ndcg_at_1\n value: 28.588\n - type: ndcg_at_10\n value: 43.511\n - type: ndcg_at_100\n value: 48.274\n - type: ndcg_at_1000\n value: 49.975\n - type: ndcg_at_3\n value: 37.319\n - type: ndcg_at_5\n value: 40.568\n - type: precision_at_1\n value: 28.588\n - type: precision_at_10\n value: 6.893000000000001\n - type: precision_at_100\n value: 0.9900000000000001\n - type: precision_at_1000\n value: 0.117\n - type: precision_at_3\n value: 16.347\n - type: precision_at_5\n value: 11.661000000000001\n - type: recall_at_1\n value: 26.27\n - type: recall_at_10\n value: 60.284000000000006\n - type: recall_at_100\n value: 81.902\n - type: recall_at_1000\n value: 94.43\n - type: recall_at_3\n value: 43.537\n - type: recall_at_5\n value: 51.475\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackMathematicaRetrieval\n revision: 90fceea13679c63fe563ded68f3b6f06e50061de\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 18.168\n - type: map_at_10\n value: 28.410000000000004\n - type: map_at_100\n value: 29.78\n - type: map_at_1000\n value: 29.892999999999997\n - type: map_at_3\n value: 25.238\n - type: map_at_5\n value: 26.96\n - type: mrr_at_1\n value: 23.507\n - type: mrr_at_10\n value: 33.382\n - type: mrr_at_100\n value: 34.404\n - type: mrr_at_1000\n value: 34.467999999999996\n - type: mrr_at_3\n value: 30.637999999999998\n - type: mrr_at_5\n value: 32.199\n - type: ndcg_at_1\n value: 23.507\n - type: ndcg_at_10\n value: 34.571000000000005\n - type: ndcg_at_100\n value: 40.663\n - type: ndcg_at_1000\n value: 43.236000000000004\n - type: ndcg_at_3\n value: 29.053\n - type: ndcg_at_5\n value: 31.563999999999997\n - type: precision_at_1\n value: 23.507\n - type: precision_at_10\n value: 6.654\n - type: precision_at_100\n value: 1.113\n - type: precision_at_1000\n value: 0.146\n - type: precision_at_3\n value: 14.427999999999999\n - type: precision_at_5\n value: 10.498000000000001\n - type: recall_at_1\n value: 18.168\n - type: recall_at_10\n value: 48.443000000000005\n - type: recall_at_100\n value: 74.47\n - type: recall_at_1000\n value: 92.494\n - type: recall_at_3\n value: 33.379999999999995\n - type: recall_at_5\n value: 39.76\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackPhysicsRetrieval\n revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 32.39\n - type: map_at_10\n value: 44.479\n - type: map_at_100\n value: 45.977000000000004\n - type: map_at_1000\n value: 46.087\n - type: map_at_3\n value: 40.976\n - type: map_at_5\n value: 43.038\n - type: mrr_at_1\n value: 40.135\n - type: mrr_at_10\n value: 50.160000000000004\n - type: mrr_at_100\n value: 51.052\n - type: mrr_at_1000\n value: 51.087\n - type: mrr_at_3\n value: 47.818\n - type: mrr_at_5\n value: 49.171\n - type: ndcg_at_1\n value: 40.135\n - type: ndcg_at_10\n value: 50.731\n - type: ndcg_at_100\n value: 56.452000000000005\n - type: ndcg_at_1000\n value: 58.123000000000005\n - type: ndcg_at_3\n value: 45.507\n - type: ndcg_at_5\n value: 48.11\n - type: precision_at_1\n value: 40.135\n - type: precision_at_10\n value: 9.192\n - type: precision_at_100\n value: 1.397\n - type: precision_at_1000\n value: 0.169\n - type: precision_at_3\n value: 21.816\n - type: precision_at_5\n value: 15.476\n - type: recall_at_1\n value: 32.39\n - type: recall_at_10\n value: 63.597\n - type: recall_at_100\n value: 86.737\n - type: recall_at_1000\n value: 97.039\n - type: recall_at_3\n value: 48.906\n - type: recall_at_5\n value: 55.659000000000006\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackProgrammersRetrieval\n revision: 6184bc1440d2dbc7612be22b50686b8826d22b32\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 28.397\n - type: map_at_10\n value: 39.871\n - type: map_at_100\n value: 41.309000000000005\n - type: map_at_1000\n value: 41.409\n - type: map_at_3\n value: 36.047000000000004\n - type: map_at_5\n value: 38.104\n - type: mrr_at_1\n value: 34.703\n - type: mrr_at_10\n value: 44.773\n - type: mrr_at_100\n value: 45.64\n - type: mrr_at_1000\n value: 45.678999999999995\n - type: mrr_at_3\n value: 41.705\n - type: mrr_at_5\n value: 43.406\n - type: ndcg_at_1\n value: 34.703\n - type: ndcg_at_10\n value: 46.271\n - type: ndcg_at_100\n value: 52.037\n - type: ndcg_at_1000\n value: 53.81700000000001\n - type: ndcg_at_3\n value: 39.966\n - type: ndcg_at_5\n value: 42.801\n - type: precision_at_1\n value: 34.703\n - type: precision_at_10\n value: 8.744\n - type: precision_at_100\n value: 1.348\n - type: precision_at_1000\n value: 0.167\n - type: precision_at_3\n value: 19.102\n - type: precision_at_5\n value: 13.836\n - type: recall_at_1\n value: 28.397\n - type: recall_at_10\n value: 60.299\n - type: recall_at_100\n value: 84.595\n - type: recall_at_1000\n value: 96.155\n - type: recall_at_3\n value: 43.065\n - type: recall_at_5\n value: 50.371\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackRetrieval\n revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 28.044333333333338\n - type: map_at_10\n value: 38.78691666666666\n - type: map_at_100\n value: 40.113\n - type: map_at_1000\n value: 40.22125\n - type: map_at_3\n value: 35.52966666666667\n - type: map_at_5\n value: 37.372749999999996\n - type: mrr_at_1\n value: 33.159083333333335\n - type: mrr_at_10\n value: 42.913583333333335\n - type: mrr_at_100\n value: 43.7845\n - type: mrr_at_1000\n value: 43.830333333333336\n - type: mrr_at_3\n value: 40.29816666666667\n - type: mrr_at_5\n value: 41.81366666666667\n - type: ndcg_at_1\n value: 33.159083333333335\n - type: ndcg_at_10\n value: 44.75750000000001\n - type: ndcg_at_100\n value: 50.13658333333334\n - type: ndcg_at_1000\n value: 52.037\n - type: ndcg_at_3\n value: 39.34258333333334\n - type: ndcg_at_5\n value: 41.93708333333333\n - type: precision_at_1\n value: 33.159083333333335\n - type: precision_at_10\n value: 7.952416666666667\n - type: precision_at_100\n value: 1.2571666666666668\n - type: precision_at_1000\n value: 0.16099999999999998\n - type: precision_at_3\n value: 18.303833333333337\n - type: precision_at_5\n value: 13.057083333333333\n - type: recall_at_1\n value: 28.044333333333338\n - type: recall_at_10\n value: 58.237249999999996\n - type: recall_at_100\n value: 81.35391666666666\n - type: recall_at_1000\n value: 94.21283333333334\n - type: recall_at_3\n value: 43.32341666666667\n - type: recall_at_5\n value: 49.94908333333333\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackStatsRetrieval\n revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 27.838\n - type: map_at_10\n value: 36.04\n - type: map_at_100\n value: 37.113\n - type: map_at_1000\n value: 37.204\n - type: map_at_3\n value: 33.585\n - type: map_at_5\n value: 34.845\n - type: mrr_at_1\n value: 30.982\n - type: mrr_at_10\n value: 39.105000000000004\n - type: mrr_at_100\n value: 39.98\n - type: mrr_at_1000\n value: 40.042\n - type: mrr_at_3\n value: 36.912\n - type: mrr_at_5\n value: 38.062000000000005\n - type: ndcg_at_1\n value: 30.982\n - type: ndcg_at_10\n value: 40.982\n - type: ndcg_at_100\n value: 46.092\n - type: ndcg_at_1000\n value: 48.25\n - type: ndcg_at_3\n value: 36.41\n - type: ndcg_at_5\n value: 38.379999999999995\n - type: precision_at_1\n value: 30.982\n - type: precision_at_10\n value: 6.534\n - type: precision_at_100\n value: 0.9820000000000001\n - type: precision_at_1000\n value: 0.124\n - type: precision_at_3\n value: 15.745999999999999\n - type: precision_at_5\n value: 10.828\n - type: recall_at_1\n value: 27.838\n - type: recall_at_10\n value: 52.971000000000004\n - type: recall_at_100\n value: 76.357\n - type: recall_at_1000\n value: 91.973\n - type: recall_at_3\n value: 40.157\n - type: recall_at_5\n value: 45.147999999999996\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackTexRetrieval\n revision: 46989137a86843e03a6195de44b09deda022eec7\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 19.059\n - type: map_at_10\n value: 27.454\n - type: map_at_100\n value: 28.736\n - type: map_at_1000\n value: 28.865000000000002\n - type: map_at_3\n value: 24.773999999999997\n - type: map_at_5\n value: 26.266000000000002\n - type: mrr_at_1\n value: 23.125\n - type: mrr_at_10\n value: 31.267\n - type: mrr_at_100\n value: 32.32\n - type: mrr_at_1000\n value: 32.394\n - type: mrr_at_3\n value: 28.894\n - type: mrr_at_5\n value: 30.281000000000002\n - type: ndcg_at_1\n value: 23.125\n - type: ndcg_at_10\n value: 32.588\n - type: ndcg_at_100\n value: 38.432\n - type: ndcg_at_1000\n value: 41.214\n - type: ndcg_at_3\n value: 27.938000000000002\n - type: ndcg_at_5\n value: 30.127\n - type: precision_at_1\n value: 23.125\n - type: precision_at_10\n value: 5.9639999999999995\n - type: precision_at_100\n value: 1.047\n - type: precision_at_1000\n value: 0.148\n - type: precision_at_3\n value: 13.294\n - type: precision_at_5\n value: 9.628\n - type: recall_at_1\n value: 19.059\n - type: recall_at_10\n value: 44.25\n - type: recall_at_100\n value: 69.948\n - type: recall_at_1000\n value: 89.35300000000001\n - type: recall_at_3\n value: 31.114000000000004\n - type: recall_at_5\n value: 36.846000000000004\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackUnixRetrieval\n revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 28.355999999999998\n - type: map_at_10\n value: 39.055\n - type: map_at_100\n value: 40.486\n - type: map_at_1000\n value: 40.571\n - type: map_at_3\n value: 35.69\n - type: map_at_5\n value: 37.605\n - type: mrr_at_1\n value: 33.302\n - type: mrr_at_10\n value: 42.986000000000004\n - type: mrr_at_100\n value: 43.957\n - type: mrr_at_1000\n value: 43.996\n - type: mrr_at_3\n value: 40.111999999999995\n - type: mrr_at_5\n value: 41.735\n - type: ndcg_at_1\n value: 33.302\n - type: ndcg_at_10\n value: 44.962999999999994\n - type: ndcg_at_100\n value: 50.917\n - type: ndcg_at_1000\n value: 52.622\n - type: ndcg_at_3\n value: 39.182\n - type: ndcg_at_5\n value: 41.939\n - type: precision_at_1\n value: 33.302\n - type: precision_at_10\n value: 7.779999999999999\n - type: precision_at_100\n value: 1.203\n - type: precision_at_1000\n value: 0.145\n - type: precision_at_3\n value: 18.035\n - type: precision_at_5\n value: 12.873000000000001\n - type: recall_at_1\n value: 28.355999999999998\n - type: recall_at_10\n value: 58.782000000000004\n - type: recall_at_100\n value: 84.02199999999999\n - type: recall_at_1000\n value: 95.511\n - type: recall_at_3\n value: 43.126999999999995\n - type: recall_at_5\n value: 50.14999999999999\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackWebmastersRetrieval\n revision: 160c094312a0e1facb97e55eeddb698c0abe3571\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 27.391\n - type: map_at_10\n value: 37.523\n - type: map_at_100\n value: 39.312000000000005\n - type: map_at_1000\n value: 39.54\n - type: map_at_3\n value: 34.231\n - type: map_at_5\n value: 36.062\n - type: mrr_at_1\n value: 32.016\n - type: mrr_at_10\n value: 41.747\n - type: mrr_at_100\n value: 42.812\n - type: mrr_at_1000\n value: 42.844\n - type: mrr_at_3\n value: 39.129999999999995\n - type: mrr_at_5\n value: 40.524\n - type: ndcg_at_1\n value: 32.016\n - type: ndcg_at_10\n value: 43.826\n - type: ndcg_at_100\n value: 50.373999999999995\n - type: ndcg_at_1000\n value: 52.318\n - type: ndcg_at_3\n value: 38.479\n - type: ndcg_at_5\n value: 40.944\n - type: precision_at_1\n value: 32.016\n - type: precision_at_10\n value: 8.280999999999999\n - type: precision_at_100\n value: 1.6760000000000002\n - type: precision_at_1000\n value: 0.25\n - type: precision_at_3\n value: 18.05\n - type: precision_at_5\n value: 13.083\n - type: recall_at_1\n value: 27.391\n - type: recall_at_10\n value: 56.928999999999995\n - type: recall_at_100\n value: 85.169\n - type: recall_at_1000\n value: 96.665\n - type: recall_at_3\n value: 42.264\n - type: recall_at_5\n value: 48.556\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CQADupstackWordpressRetrieval\n revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4\n split: test\n type: BeIR/cqadupstack\n metrics:\n - type: map_at_1\n value: 18.398\n - type: map_at_10\n value: 27.929\n - type: map_at_100\n value: 29.032999999999998\n - type: map_at_1000\n value: 29.126\n - type: map_at_3\n value: 25.070999999999998\n - type: map_at_5\n value: 26.583000000000002\n - type: mrr_at_1\n value: 19.963\n - type: mrr_at_10\n value: 29.997\n - type: mrr_at_100\n value: 30.9\n - type: mrr_at_1000\n value: 30.972\n - type: mrr_at_3\n value: 27.264\n - type: mrr_at_5\n value: 28.826\n - type: ndcg_at_1\n value: 19.963\n - type: ndcg_at_10\n value: 33.678999999999995\n - type: ndcg_at_100\n value: 38.931\n - type: ndcg_at_1000\n value: 41.379\n - type: ndcg_at_3\n value: 28.000000000000004\n - type: ndcg_at_5\n value: 30.637999999999998\n - type: precision_at_1\n value: 19.963\n - type: precision_at_10\n value: 5.7299999999999995\n - type: precision_at_100\n value: 0.902\n - type: precision_at_1000\n value: 0.122\n - type: precision_at_3\n value: 12.631\n - type: precision_at_5\n value: 9.057\n - type: recall_at_1\n value: 18.398\n - type: recall_at_10\n value: 49.254\n - type: recall_at_100\n value: 73.182\n - type: recall_at_1000\n value: 91.637\n - type: recall_at_3\n value: 34.06\n - type: recall_at_5\n value: 40.416000000000004\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB ClimateFEVER\n revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380\n split: test\n type: mteb/climate-fever\n metrics:\n - type: map_at_1\n value: 19.681\n - type: map_at_10\n value: 32.741\n - type: map_at_100\n value: 34.811\n - type: map_at_1000\n value: 35.003\n - type: map_at_3\n value: 27.697\n - type: map_at_5\n value: 30.372\n - type: mrr_at_1\n value: 44.951\n - type: mrr_at_10\n value: 56.34400000000001\n - type: mrr_at_100\n value: 56.961\n - type: mrr_at_1000\n value: 56.987\n - type: mrr_at_3\n value: 53.681\n - type: mrr_at_5\n value: 55.407\n - type: ndcg_at_1\n value: 44.951\n - type: ndcg_at_10\n value: 42.905\n - type: ndcg_at_100\n value: 49.95\n - type: ndcg_at_1000\n value: 52.917\n - type: ndcg_at_3\n value: 36.815\n - type: ndcg_at_5\n value: 38.817\n - type: precision_at_1\n value: 44.951\n - type: precision_at_10\n value: 12.989999999999998\n - type: precision_at_100\n value: 2.068\n - type: precision_at_1000\n value: 0.263\n - type: precision_at_3\n value: 27.275\n - type: precision_at_5\n value: 20.365\n - type: recall_at_1\n value: 19.681\n - type: recall_at_10\n value: 48.272999999999996\n - type: recall_at_100\n value: 71.87400000000001\n - type: recall_at_1000\n value: 87.929\n - type: recall_at_3\n value: 32.653999999999996\n - type: recall_at_5\n value: 39.364\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB DBPedia\n revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659\n split: test\n type: mteb/dbpedia\n metrics:\n - type: map_at_1\n value: 10.231\n - type: map_at_10\n value: 22.338\n - type: map_at_100\n value: 31.927\n - type: map_at_1000\n value: 33.87\n - type: map_at_3\n value: 15.559999999999999\n - type: map_at_5\n value: 18.239\n - type: mrr_at_1\n value: 75.0\n - type: mrr_at_10\n value: 81.303\n - type: mrr_at_100\n value: 81.523\n - type: mrr_at_1000\n value: 81.53\n - type: mrr_at_3\n value: 80.083\n - type: mrr_at_5\n value: 80.758\n - type: ndcg_at_1\n value: 64.625\n - type: ndcg_at_10\n value: 48.687000000000005\n - type: ndcg_at_100\n value: 52.791\n - type: ndcg_at_1000\n value: 60.041999999999994\n - type: ndcg_at_3\n value: 53.757999999999996\n - type: ndcg_at_5\n value: 50.76500000000001\n - type: precision_at_1\n value: 75.0\n - type: precision_at_10\n value: 38.3\n - type: precision_at_100\n value: 12.025\n - type: precision_at_1000\n value: 2.3970000000000002\n - type: precision_at_3\n value: 55.417\n - type: precision_at_5\n value: 47.5\n - type: recall_at_1\n value: 10.231\n - type: recall_at_10\n value: 27.697\n - type: recall_at_100\n value: 57.409\n - type: recall_at_1000\n value: 80.547\n - type: recall_at_3\n value: 16.668\n - type: recall_at_5\n value: 20.552\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB EmotionClassification\n revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37\n split: test\n type: mteb/emotion\n metrics:\n - type: accuracy\n value: 61.365\n - type: f1\n value: 56.7540827912991\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB FEVER\n revision: bea83ef9e8fb933d90a2f1d5515737465d613e12\n split: test\n type: mteb/fever\n metrics:\n - type: map_at_1\n value: 83.479\n - type: map_at_10\n value: 88.898\n - type: map_at_100\n value: 89.11\n - type: map_at_1000\n value: 89.12400000000001\n - type: map_at_3\n value: 88.103\n - type: map_at_5\n value: 88.629\n - type: mrr_at_1\n value: 89.934\n - type: mrr_at_10\n value: 93.91000000000001\n - type: mrr_at_100\n value: 93.937\n - type: mrr_at_1000\n value: 93.938\n - type: mrr_at_3\n value: 93.62700000000001\n - type: mrr_at_5\n value: 93.84599999999999\n - type: ndcg_at_1\n value: 89.934\n - type: ndcg_at_10\n value: 91.574\n - type: ndcg_at_100\n value: 92.238\n - type: ndcg_at_1000\n value: 92.45\n - type: ndcg_at_3\n value: 90.586\n - type: ndcg_at_5\n value: 91.16300000000001\n - type: precision_at_1\n value: 89.934\n - type: precision_at_10\n value: 10.555\n - type: precision_at_100\n value: 1.1159999999999999\n - type: precision_at_1000\n value: 0.11499999999999999\n - type: precision_at_3\n value: 33.588\n - type: precision_at_5\n value: 20.642\n - type: recall_at_1\n value: 83.479\n - type: recall_at_10\n value: 94.971\n - type: recall_at_100\n value: 97.397\n - type: recall_at_1000\n value: 98.666\n - type: recall_at_3\n value: 92.24799999999999\n - type: recall_at_5\n value: 93.797\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB FiQA2018\n revision: 27a168819829fe9bcd655c2df245fb19452e8e06\n split: test\n type: mteb/fiqa\n metrics:\n - type: map_at_1\n value: 27.16\n - type: map_at_10\n value: 45.593\n - type: map_at_100\n value: 47.762\n - type: map_at_1000\n value: 47.899\n - type: map_at_3\n value: 39.237\n - type: map_at_5\n value: 42.970000000000006\n - type: mrr_at_1\n value: 52.623\n - type: mrr_at_10\n value: 62.637\n - type: mrr_at_100\n value: 63.169\n - type: mrr_at_1000\n value: 63.185\n - type: mrr_at_3\n value: 59.928000000000004\n - type: mrr_at_5\n value: 61.702999999999996\n - type: ndcg_at_1\n value: 52.623\n - type: ndcg_at_10\n value: 54.701\n - type: ndcg_at_100\n value: 61.263\n - type: ndcg_at_1000\n value: 63.134\n - type: ndcg_at_3\n value: 49.265\n - type: ndcg_at_5\n value: 51.665000000000006\n - type: precision_at_1\n value: 52.623\n - type: precision_at_10\n value: 15.185\n - type: precision_at_100\n value: 2.202\n - type: precision_at_1000\n value: 0.254\n - type: precision_at_3\n value: 32.767\n - type: precision_at_5\n value: 24.722\n - type: recall_at_1\n value: 27.16\n - type: recall_at_10\n value: 63.309000000000005\n - type: recall_at_100\n value: 86.722\n - type: recall_at_1000\n value: 97.505\n - type: recall_at_3\n value: 45.045\n - type: recall_at_5\n value: 54.02400000000001\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB HotpotQA\n revision: ab518f4d6fcca38d87c25209f94beba119d02014\n split: test\n type: mteb/hotpotqa\n metrics:\n - type: map_at_1\n value: 42.573\n - type: map_at_10\n value: 59.373\n - type: map_at_100\n value: 60.292\n - type: map_at_1000\n value: 60.358999999999995\n - type: map_at_3\n value: 56.159000000000006\n - type: map_at_5\n value: 58.123999999999995\n - type: mrr_at_1\n value: 85.14500000000001\n - type: mrr_at_10\n value: 89.25999999999999\n - type: mrr_at_100\n value: 89.373\n - type: mrr_at_1000\n value: 89.377\n - type: mrr_at_3\n value: 88.618\n - type: mrr_at_5\n value: 89.036\n - type: ndcg_at_1\n value: 85.14500000000001\n - type: ndcg_at_10\n value: 68.95\n - type: ndcg_at_100\n value: 71.95\n - type: ndcg_at_1000\n value: 73.232\n - type: ndcg_at_3\n value: 64.546\n - type: ndcg_at_5\n value: 66.945\n - type: precision_at_1\n value: 85.14500000000001\n - type: precision_at_10\n value: 13.865\n - type: precision_at_100\n value: 1.619\n - type: precision_at_1000\n value: 0.179\n - type: precision_at_3\n value: 39.703\n - type: precision_at_5\n value: 25.718000000000004\n - type: recall_at_1\n value: 42.573\n - type: recall_at_10\n value: 69.325\n - type: recall_at_100\n value: 80.932\n - type: recall_at_1000\n value: 89.446\n - type: recall_at_3\n value: 59.553999999999995\n - type: recall_at_5\n value: 64.294\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB ImdbClassification\n revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7\n split: test\n type: mteb/imdb\n metrics:\n - type: accuracy\n value: 95.8336\n - type: ap\n value: 93.78862962194073\n - type: f1\n value: 95.83192650728371\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB MSMARCO\n revision: c5a29a104738b98a9e76336939199e264163d4a0\n split: dev\n type: mteb/msmarco\n metrics:\n - type: map_at_1\n value: 23.075000000000003\n - type: map_at_10\n value: 36.102000000000004\n - type: map_at_100\n value: 37.257\n - type: map_at_1000\n value: 37.3\n - type: map_at_3\n value: 32.144\n - type: map_at_5\n value: 34.359\n - type: mrr_at_1\n value: 23.711\n - type: mrr_at_10\n value: 36.671\n - type: mrr_at_100\n value: 37.763999999999996\n - type: mrr_at_1000\n value: 37.801\n - type: mrr_at_3\n value: 32.775\n - type: mrr_at_5\n value: 34.977000000000004\n - type: ndcg_at_1\n value: 23.711\n - type: ndcg_at_10\n value: 43.361\n - type: ndcg_at_100\n value: 48.839\n - type: ndcg_at_1000\n value: 49.88\n - type: ndcg_at_3\n value: 35.269\n - type: ndcg_at_5\n value: 39.224\n - type: precision_at_1\n value: 23.711\n - type: precision_at_10\n value: 6.866999999999999\n - type: precision_at_100\n value: 0.96\n - type: precision_at_1000\n value: 0.105\n - type: precision_at_3\n value: 15.096000000000002\n - type: precision_at_5\n value: 11.083\n - type: recall_at_1\n value: 23.075000000000003\n - type: recall_at_10\n value: 65.756\n - type: recall_at_100\n value: 90.88199999999999\n - type: recall_at_1000\n value: 98.739\n - type: recall_at_3\n value: 43.691\n - type: recall_at_5\n value: 53.15800000000001\n task:\n type: Retrieval\n - dataset:\n config: en\n name: MTEB MTOPDomainClassification (en)\n revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf\n split: test\n type: mteb/mtop_domain\n metrics:\n - type: accuracy\n value: 97.69493844049248\n - type: f1\n value: 97.55048089616261\n task:\n type: Classification\n - dataset:\n config: en\n name: MTEB MTOPIntentClassification (en)\n revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba\n split: test\n type: mteb/mtop_intent\n metrics:\n - type: accuracy\n value: 88.75968992248062\n - type: f1\n value: 72.26321223399123\n task:\n type: Classification\n - dataset:\n config: en\n name: MTEB MassiveIntentClassification (en)\n revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7\n split: test\n type: mteb/amazon_massive_intent\n metrics:\n - type: accuracy\n value: 82.40080699394754\n - type: f1\n value: 79.62590029057968\n task:\n type: Classification\n - dataset:\n config: en\n name: MTEB MassiveScenarioClassification (en)\n revision: 7d571f92784cd94a019292a1f45445077d0ef634\n split: test\n type: mteb/amazon_massive_scenario\n metrics:\n - type: accuracy\n value: 84.49562878278414\n - type: f1\n value: 84.0040193313333\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB MedrxivClusteringP2P\n revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73\n split: test\n type: mteb/medrxiv-clustering-p2p\n metrics:\n - type: v_measure\n value: 39.386760057101945\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB MedrxivClusteringS2S\n revision: 35191c8c0dca72d8ff3efcd72aa802307d469663\n split: test\n type: mteb/medrxiv-clustering-s2s\n metrics:\n - type: v_measure\n value: 37.89687154075537\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB MindSmallReranking\n revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69\n split: test\n type: mteb/mind_small\n metrics:\n - type: map\n value: 33.94151656057482\n - type: mrr\n value: 35.32684700746953\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB NFCorpus\n revision: ec0fa4fe99da2ff19ca1214b7966684033a58814\n split: test\n type: mteb/nfcorpus\n metrics:\n - type: map_at_1\n value: 6.239999999999999\n - type: map_at_10\n value: 14.862\n - type: map_at_100\n value: 18.955\n - type: map_at_1000\n value: 20.694000000000003\n - type: map_at_3\n value: 10.683\n - type: map_at_5\n value: 12.674\n - type: mrr_at_1\n value: 50.15500000000001\n - type: mrr_at_10\n value: 59.697\n - type: mrr_at_100\n value: 60.095\n - type: mrr_at_1000\n value: 60.129999999999995\n - type: mrr_at_3\n value: 58.35900000000001\n - type: mrr_at_5\n value: 58.839\n - type: ndcg_at_1\n value: 48.452\n - type: ndcg_at_10\n value: 39.341\n - type: ndcg_at_100\n value: 35.866\n - type: ndcg_at_1000\n value: 45.111000000000004\n - type: ndcg_at_3\n value: 44.527\n - type: ndcg_at_5\n value: 42.946\n - type: precision_at_1\n value: 50.15500000000001\n - type: precision_at_10\n value: 29.536\n - type: precision_at_100\n value: 9.142\n - type: precision_at_1000\n value: 2.2849999999999997\n - type: precision_at_3\n value: 41.899\n - type: precision_at_5\n value: 37.647000000000006\n - type: recall_at_1\n value: 6.239999999999999\n - type: recall_at_10\n value: 19.278000000000002\n - type: recall_at_100\n value: 36.074\n - type: recall_at_1000\n value: 70.017\n - type: recall_at_3\n value: 12.066\n - type: recall_at_5\n value: 15.254000000000001\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB NQ\n revision: b774495ed302d8c44a3a7ea25c90dbce03968f31\n split: test\n type: mteb/nq\n metrics:\n - type: map_at_1\n value: 39.75\n - type: map_at_10\n value: 56.443\n - type: map_at_100\n value: 57.233999999999995\n - type: map_at_1000\n value: 57.249\n - type: map_at_3\n value: 52.032999999999994\n - type: map_at_5\n value: 54.937999999999995\n - type: mrr_at_1\n value: 44.728\n - type: mrr_at_10\n value: 58.939\n - type: mrr_at_100\n value: 59.489000000000004\n - type: mrr_at_1000\n value: 59.499\n - type: mrr_at_3\n value: 55.711999999999996\n - type: mrr_at_5\n value: 57.89\n - type: ndcg_at_1\n value: 44.728\n - type: ndcg_at_10\n value: 63.998999999999995\n - type: ndcg_at_100\n value: 67.077\n - type: ndcg_at_1000\n value: 67.40899999999999\n - type: ndcg_at_3\n value: 56.266000000000005\n - type: ndcg_at_5\n value: 60.88\n - type: precision_at_1\n value: 44.728\n - type: precision_at_10\n value: 10.09\n - type: precision_at_100\n value: 1.1809999999999998\n - type: precision_at_1000\n value: 0.121\n - type: precision_at_3\n value: 25.145\n - type: precision_at_5\n value: 17.822\n - type: recall_at_1\n value: 39.75\n - type: recall_at_10\n value: 84.234\n - type: recall_at_100\n value: 97.055\n - type: recall_at_1000\n value: 99.517\n - type: recall_at_3\n value: 64.851\n - type: recall_at_5\n value: 75.343\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB QuoraRetrieval\n revision: None\n split: test\n type: mteb/quora\n metrics:\n - type: map_at_1\n value: 72.085\n - type: map_at_10\n value: 86.107\n - type: map_at_100\n value: 86.727\n - type: map_at_1000\n value: 86.74\n - type: map_at_3\n value: 83.21\n - type: map_at_5\n value: 85.06\n - type: mrr_at_1\n value: 82.94\n - type: mrr_at_10\n value: 88.845\n - type: mrr_at_100\n value: 88.926\n - type: mrr_at_1000\n value: 88.927\n - type: mrr_at_3\n value: 87.993\n - type: mrr_at_5\n value: 88.62299999999999\n - type: ndcg_at_1\n value: 82.97\n - type: ndcg_at_10\n value: 89.645\n - type: ndcg_at_100\n value: 90.717\n - type: ndcg_at_1000\n value: 90.78\n - type: ndcg_at_3\n value: 86.99900000000001\n - type: ndcg_at_5\n value: 88.52600000000001\n - type: precision_at_1\n value: 82.97\n - type: precision_at_10\n value: 13.569\n - type: precision_at_100\n value: 1.539\n - type: precision_at_1000\n value: 0.157\n - type: precision_at_3\n value: 38.043\n - type: precision_at_5\n value: 24.992\n - type: recall_at_1\n value: 72.085\n - type: recall_at_10\n value: 96.262\n - type: recall_at_100\n value: 99.77000000000001\n - type: recall_at_1000\n value: 99.997\n - type: recall_at_3\n value: 88.652\n - type: recall_at_5\n value: 93.01899999999999\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB RedditClustering\n revision: 24640382cdbf8abc73003fb0fa6d111a705499eb\n split: test\n type: mteb/reddit-clustering\n metrics:\n - type: v_measure\n value: 55.82153952668092\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB RedditClusteringP2P\n revision: 282350215ef01743dc01b456c7f5241fa8937f16\n split: test\n type: mteb/reddit-clustering-p2p\n metrics:\n - type: v_measure\n value: 62.094465801879295\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB SCIDOCS\n revision: None\n split: test\n type: mteb/scidocs\n metrics:\n - type: map_at_1\n value: 5.688\n - type: map_at_10\n value: 15.201999999999998\n - type: map_at_100\n value: 18.096\n - type: map_at_1000\n value: 18.481\n - type: map_at_3\n value: 10.734\n - type: map_at_5\n value: 12.94\n - type: mrr_at_1\n value: 28.000000000000004\n - type: mrr_at_10\n value: 41.101\n - type: mrr_at_100\n value: 42.202\n - type: mrr_at_1000\n value: 42.228\n - type: mrr_at_3\n value: 37.683\n - type: mrr_at_5\n value: 39.708\n - type: ndcg_at_1\n value: 28.000000000000004\n - type: ndcg_at_10\n value: 24.976000000000003\n - type: ndcg_at_100\n value: 35.129\n - type: ndcg_at_1000\n value: 40.77\n - type: ndcg_at_3\n value: 23.787\n - type: ndcg_at_5\n value: 20.816000000000003\n - type: precision_at_1\n value: 28.000000000000004\n - type: precision_at_10\n value: 13.04\n - type: precision_at_100\n value: 2.761\n - type: precision_at_1000\n value: 0.41000000000000003\n - type: precision_at_3\n value: 22.6\n - type: precision_at_5\n value: 18.52\n - type: recall_at_1\n value: 5.688\n - type: recall_at_10\n value: 26.43\n - type: recall_at_100\n value: 56.02\n - type: recall_at_1000\n value: 83.21\n - type: recall_at_3\n value: 13.752\n - type: recall_at_5\n value: 18.777\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB SICK-R\n revision: a6ea5a8cab320b040a23452cc28066d9beae2cee\n split: test\n type: mteb/sickr-sts\n metrics:\n - type: cos_sim_pearson\n value: 85.15084859283178\n - type: cos_sim_spearman\n value: 80.49030614009419\n - type: euclidean_pearson\n value: 81.84574978672468\n - type: euclidean_spearman\n value: 79.89787150656818\n - type: manhattan_pearson\n value: 81.63076538567131\n - type: manhattan_spearman\n value: 79.69867352121841\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB STS12\n revision: a0d554a64d88156834ff5ae9920b964011b16384\n split: test\n type: mteb/sts12-sts\n metrics:\n - type: cos_sim_pearson\n value: 84.64097921490992\n - type: cos_sim_spearman\n value: 77.25370084896514\n - type: euclidean_pearson\n value: 82.71210826468788\n - type: euclidean_spearman\n value: 78.50445584994826\n - type: manhattan_pearson\n value: 82.92580164330298\n - type: manhattan_spearman\n value: 78.69686891301019\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB STS13\n revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca\n split: test\n type: mteb/sts13-sts\n metrics:\n - type: cos_sim_pearson\n value: 87.24596417308994\n - type: cos_sim_spearman\n value: 87.79454220555091\n - type: euclidean_pearson\n value: 87.40242561671164\n - type: euclidean_spearman\n value: 88.25955597373556\n - type: manhattan_pearson\n value: 87.25160240485849\n - type: manhattan_spearman\n value: 88.155794979818\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB STS14\n revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375\n split: test\n type: mteb/sts14-sts\n metrics:\n - type: cos_sim_pearson\n value: 84.44914233422564\n - type: cos_sim_spearman\n value: 82.91015471820322\n - type: euclidean_pearson\n value: 84.7206656630327\n - type: euclidean_spearman\n value: 83.86408872059216\n - type: manhattan_pearson\n value: 84.72816725158454\n - type: manhattan_spearman\n value: 84.01603388572788\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB STS15\n revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3\n split: test\n type: mteb/sts15-sts\n metrics:\n - type: cos_sim_pearson\n value: 87.6168026237477\n - type: cos_sim_spearman\n value: 88.45414278092397\n - type: euclidean_pearson\n value: 88.57023240882022\n - type: euclidean_spearman\n value: 89.04102190922094\n - type: manhattan_pearson\n value: 88.66695535796354\n - type: manhattan_spearman\n value: 89.19898476680969\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB STS16\n revision: 4d8694f8f0e0100860b497b999b3dbed754a0513\n split: test\n type: mteb/sts16-sts\n metrics:\n - type: cos_sim_pearson\n value: 84.27925826089424\n - type: cos_sim_spearman\n value: 85.45291099550461\n - type: euclidean_pearson\n value: 83.63853036580834\n - type: euclidean_spearman\n value: 84.33468035821484\n - type: manhattan_pearson\n value: 83.72778773251596\n - type: manhattan_spearman\n value: 84.51583132445376\n task:\n type: STS\n - dataset:\n config: en-en\n name: MTEB STS17 (en-en)\n revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d\n split: test\n type: mteb/sts17-crosslingual-sts\n metrics:\n - type: cos_sim_pearson\n value: 89.67375185692552\n - type: cos_sim_spearman\n value: 90.32542469203855\n - type: euclidean_pearson\n value: 89.63513717951847\n - type: euclidean_spearman\n value: 89.87760271003745\n - type: manhattan_pearson\n value: 89.28381452982924\n - type: manhattan_spearman\n value: 89.53568197785721\n task:\n type: STS\n - dataset:\n config: en\n name: MTEB STS22 (en)\n revision: eea2b4fe26a775864c896887d910b76a8098ad3f\n split: test\n type: mteb/sts22-crosslingual-sts\n metrics:\n - type: cos_sim_pearson\n value: 66.24644693819846\n - type: cos_sim_spearman\n value: 66.09889420525377\n - type: euclidean_pearson\n value: 63.72551583520747\n - type: euclidean_spearman\n value: 63.01385470780679\n - type: manhattan_pearson\n value: 64.09258157214097\n - type: manhattan_spearman\n value: 63.080517752822594\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB STSBenchmark\n revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831\n split: test\n type: mteb/stsbenchmark-sts\n metrics:\n - type: cos_sim_pearson\n value: 86.27321463839989\n - type: cos_sim_spearman\n value: 86.37572865993327\n - type: euclidean_pearson\n value: 86.36268020198149\n - type: euclidean_spearman\n value: 86.31089339478922\n - type: manhattan_pearson\n value: 86.4260445761947\n - type: manhattan_spearman\n value: 86.45885895320457\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB SciDocsRR\n revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab\n split: test\n type: mteb/scidocs-reranking\n metrics:\n - type: map\n value: 86.52456702387798\n - type: mrr\n value: 96.34556529164372\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB SciFact\n revision: 0228b52cf27578f30900b9e5271d331663a030d7\n split: test\n type: mteb/scifact\n metrics:\n - type: map_at_1\n value: 61.99400000000001\n - type: map_at_10\n value: 73.38799999999999\n - type: map_at_100\n value: 73.747\n - type: map_at_1000\n value: 73.75\n - type: map_at_3\n value: 70.04599999999999\n - type: map_at_5\n value: 72.095\n - type: mrr_at_1\n value: 65.0\n - type: mrr_at_10\n value: 74.42800000000001\n - type: mrr_at_100\n value: 74.722\n - type: mrr_at_1000\n value: 74.725\n - type: mrr_at_3\n value: 72.056\n - type: mrr_at_5\n value: 73.60600000000001\n - type: ndcg_at_1\n value: 65.0\n - type: ndcg_at_10\n value: 78.435\n - type: ndcg_at_100\n value: 79.922\n - type: ndcg_at_1000\n value: 80.00500000000001\n - type: ndcg_at_3\n value: 73.05199999999999\n - type: ndcg_at_5\n value: 75.98\n - type: precision_at_1\n value: 65.0\n - type: precision_at_10\n value: 10.5\n - type: precision_at_100\n value: 1.123\n - type: precision_at_1000\n value: 0.11299999999999999\n - type: precision_at_3\n value: 28.555999999999997\n - type: precision_at_5\n value: 19.0\n - type: recall_at_1\n value: 61.99400000000001\n - type: recall_at_10\n value: 92.72200000000001\n - type: recall_at_100\n value: 99.333\n - type: recall_at_1000\n value: 100.0\n - type: recall_at_3\n value: 78.739\n - type: recall_at_5\n value: 85.828\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB SprintDuplicateQuestions\n revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46\n split: test\n type: mteb/sprintduplicatequestions-pairclassification\n metrics:\n - type: cos_sim_accuracy\n value: 99.79009900990098\n - type: cos_sim_ap\n value: 95.3203137438653\n - type: cos_sim_f1\n value: 89.12386706948641\n - type: cos_sim_precision\n value: 89.75659229208925\n - type: cos_sim_recall\n value: 88.5\n - type: dot_accuracy\n value: 99.67821782178218\n - type: dot_ap\n value: 89.94069840000675\n - type: dot_f1\n value: 83.45902463549521\n - type: dot_precision\n value: 83.9231547017189\n - type: dot_recall\n value: 83.0\n - type: euclidean_accuracy\n value: 99.78613861386138\n - type: euclidean_ap\n value: 95.10648259135526\n - type: euclidean_f1\n value: 88.77338877338877\n - type: euclidean_precision\n value: 92.42424242424242\n - type: euclidean_recall\n value: 85.39999999999999\n - type: manhattan_accuracy\n value: 99.7950495049505\n - type: manhattan_ap\n value: 95.29987661320946\n - type: manhattan_f1\n value: 89.21313183949972\n - type: manhattan_precision\n value: 93.14472252448314\n - type: manhattan_recall\n value: 85.6\n - type: max_accuracy\n value: 99.7950495049505\n - type: max_ap\n value: 95.3203137438653\n - type: max_f1\n value: 89.21313183949972\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB StackExchangeClustering\n revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259\n split: test\n type: mteb/stackexchange-clustering\n metrics:\n - type: v_measure\n value: 67.65446577183913\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB StackExchangeClusteringP2P\n revision: 815ca46b2622cec33ccafc3735d572c266efdb44\n split: test\n type: mteb/stackexchange-clustering-p2p\n metrics:\n - type: v_measure\n value: 46.30749237193961\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB StackOverflowDupQuestions\n revision: e185fbe320c72810689fc5848eb6114e1ef5ec69\n split: test\n type: mteb/stackoverflowdupquestions-reranking\n metrics:\n - type: map\n value: 54.91481849959949\n - type: mrr\n value: 55.853506175197346\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB SummEval\n revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c\n split: test\n type: mteb/summeval\n metrics:\n - type: cos_sim_pearson\n value: 30.08196549170419\n - type: cos_sim_spearman\n value: 31.16661390597077\n - type: dot_pearson\n value: 29.892258410943466\n - type: dot_spearman\n value: 30.51328811965085\n task:\n type: Summarization\n - dataset:\n config: default\n name: MTEB TRECCOVID\n revision: None\n split: test\n type: mteb/trec-covid\n metrics:\n - type: map_at_1\n value: 0.23900000000000002\n - type: map_at_10\n value: 2.173\n - type: map_at_100\n value: 14.24\n - type: map_at_1000\n value: 35.309000000000005\n - type: map_at_3\n value: 0.7100000000000001\n - type: map_at_5\n value: 1.163\n - type: mrr_at_1\n value: 92.0\n - type: mrr_at_10\n value: 96.0\n - type: mrr_at_100\n value: 96.0\n - type: mrr_at_1000\n value: 96.0\n - type: mrr_at_3\n value: 96.0\n - type: mrr_at_5\n value: 96.0\n - type: ndcg_at_1\n value: 90.0\n - type: ndcg_at_10\n value: 85.382\n - type: ndcg_at_100\n value: 68.03\n - type: ndcg_at_1000\n value: 61.021\n - type: ndcg_at_3\n value: 89.765\n - type: ndcg_at_5\n value: 88.444\n - type: precision_at_1\n value: 92.0\n - type: precision_at_10\n value: 88.0\n - type: precision_at_100\n value: 70.02000000000001\n - type: precision_at_1000\n value: 26.984\n - type: precision_at_3\n value: 94.0\n - type: precision_at_5\n value: 92.80000000000001\n - type: recall_at_1\n value: 0.23900000000000002\n - type: recall_at_10\n value: 2.313\n - type: recall_at_100\n value: 17.049\n - type: recall_at_1000\n value: 57.489999999999995\n - type: recall_at_3\n value: 0.737\n - type: recall_at_5\n value: 1.221\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB Touche2020\n revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f\n split: test\n type: mteb/touche2020\n metrics:\n - type: map_at_1\n value: 2.75\n - type: map_at_10\n value: 11.29\n - type: map_at_100\n value: 18.032999999999998\n - type: map_at_1000\n value: 19.746\n - type: map_at_3\n value: 6.555\n - type: map_at_5\n value: 8.706999999999999\n - type: mrr_at_1\n value: 34.694\n - type: mrr_at_10\n value: 50.55\n - type: mrr_at_100\n value: 51.659\n - type: mrr_at_1000\n value: 51.659\n - type: mrr_at_3\n value: 47.278999999999996\n - type: mrr_at_5\n value: 49.728\n - type: ndcg_at_1\n value: 32.653\n - type: ndcg_at_10\n value: 27.894000000000002\n - type: ndcg_at_100\n value: 39.769\n - type: ndcg_at_1000\n value: 51.495999999999995\n - type: ndcg_at_3\n value: 32.954\n - type: ndcg_at_5\n value: 31.502999999999997\n - type: precision_at_1\n value: 34.694\n - type: precision_at_10\n value: 23.265\n - type: precision_at_100\n value: 7.898\n - type: precision_at_1000\n value: 1.58\n - type: precision_at_3\n value: 34.694\n - type: precision_at_5\n value: 31.429000000000002\n - type: recall_at_1\n value: 2.75\n - type: recall_at_10\n value: 16.953\n - type: recall_at_100\n value: 48.68\n - type: recall_at_1000\n value: 85.18599999999999\n - type: recall_at_3\n value: 7.710999999999999\n - type: recall_at_5\n value: 11.484\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB ToxicConversationsClassification\n revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c\n split: test\n type: mteb/toxic_conversations_50k\n metrics:\n - type: accuracy\n value: 82.66099999999999\n - type: ap\n value: 25.555698090238337\n - type: f1\n value: 66.48402012461622\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB TweetSentimentExtractionClassification\n revision: d604517c81ca91fe16a244d1248fc021f9ecee7a\n split: test\n type: mteb/tweet_sentiment_extraction\n metrics:\n - type: accuracy\n value: 72.94567062818335\n - type: f1\n value: 73.28139189595674\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB TwentyNewsgroupsClustering\n revision: 6125ec4e24fa026cec8a478383ee943acfbd5449\n split: test\n type: mteb/twentynewsgroups-clustering\n metrics:\n - type: v_measure\n value: 49.581627240203474\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB TwitterSemEval2015\n revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1\n split: test\n type: mteb/twittersemeval2015-pairclassification\n metrics:\n - type: cos_sim_accuracy\n value: 87.78089050485785\n - type: cos_sim_ap\n value: 79.64487116574168\n - type: cos_sim_f1\n value: 72.46563021970964\n - type: cos_sim_precision\n value: 70.62359128474831\n - type: cos_sim_recall\n value: 74.40633245382587\n - type: dot_accuracy\n value: 86.2609524944865\n - type: dot_ap\n value: 75.513046857613\n - type: dot_f1\n value: 68.58213616489695\n - type: dot_precision\n value: 65.12455516014235\n - type: dot_recall\n value: 72.42744063324538\n - type: euclidean_accuracy\n value: 87.6080348095607\n - type: euclidean_ap\n value: 79.00204933649795\n - type: euclidean_f1\n value: 72.14495342605589\n - type: euclidean_precision\n value: 69.85421299728193\n - type: euclidean_recall\n value: 74.5910290237467\n - type: manhattan_accuracy\n value: 87.59611372712642\n - type: manhattan_ap\n value: 78.78523756706264\n - type: manhattan_f1\n value: 71.86499137718648\n - type: manhattan_precision\n value: 67.39833641404806\n - type: manhattan_recall\n value: 76.96569920844327\n - type: max_accuracy\n value: 87.78089050485785\n - type: max_ap\n value: 79.64487116574168\n - type: max_f1\n value: 72.46563021970964\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB TwitterURLCorpus\n revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf\n split: test\n type: mteb/twitterurlcorpus-pairclassification\n metrics:\n - type: cos_sim_accuracy\n value: 89.98719292117825\n - type: cos_sim_ap\n value: 87.58146137353202\n - type: cos_sim_f1\n value: 80.28543232369239\n - type: cos_sim_precision\n value: 79.1735289714029\n - type: cos_sim_recall\n value: 81.42901139513397\n - type: dot_accuracy\n value: 88.9199363526992\n - type: dot_ap\n value: 84.98499998630417\n - type: dot_f1\n value: 78.21951400757969\n - type: dot_precision\n value: 75.58523624874336\n - type: dot_recall\n value: 81.04404065291038\n - type: euclidean_accuracy\n value: 89.77374160748244\n - type: euclidean_ap\n value: 87.35151562835209\n - type: euclidean_f1\n value: 79.92160922940393\n - type: euclidean_precision\n value: 76.88531587933979\n - type: euclidean_recall\n value: 83.20757622420696\n - type: manhattan_accuracy\n value: 89.72717041176699\n - type: manhattan_ap\n value: 87.34065592142515\n - type: manhattan_f1\n value: 79.85603419187943\n - type: manhattan_precision\n value: 77.82243332115455\n - type: manhattan_recall\n value: 81.99876809362489\n - type: max_accuracy\n value: 89.98719292117825\n - type: max_ap\n value: 87.58146137353202\n - type: max_f1\n value: 80.28543232369239\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB AFQMC\n revision: b44c3b011063adb25877c13823db83bb193913c4\n split: validation\n type: C-MTEB/AFQMC\n metrics:\n - type: cos_sim_pearson\n value: 53.45954203592337\n - type: cos_sim_spearman\n value: 58.42154680418638\n - type: euclidean_pearson\n value: 56.41543791722753\n - type: euclidean_spearman\n value: 58.39328016640146\n - type: manhattan_pearson\n value: 56.318510356833876\n - type: manhattan_spearman\n value: 58.28423447818184\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB ATEC\n revision: 0f319b1142f28d00e055a6770f3f726ae9b7d865\n split: test\n type: C-MTEB/ATEC\n metrics:\n - type: cos_sim_pearson\n value: 50.78356460675945\n - type: cos_sim_spearman\n value: 55.6530411663269\n - type: euclidean_pearson\n value: 56.50763660417816\n - type: euclidean_spearman\n value: 55.733823335669065\n - type: manhattan_pearson\n value: 56.45323093512866\n - type: manhattan_spearman\n value: 55.63248619032702\n task:\n type: STS\n - dataset:\n config: zh\n name: MTEB AmazonReviewsClassification (zh)\n revision: 1399c76144fd37290681b995c656ef9b2e06e26d\n split: test\n type: mteb/amazon_reviews_multi\n metrics:\n - type: accuracy\n value: 47.209999999999994\n - type: f1\n value: 46.08892432018655\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB BQ\n revision: e3dda5e115e487b39ec7e618c0c6a29137052a55\n split: test\n type: C-MTEB/BQ\n metrics:\n - type: cos_sim_pearson\n value: 70.25573992001478\n - type: cos_sim_spearman\n value: 73.85247134951433\n - type: euclidean_pearson\n value: 72.60033082168442\n - type: euclidean_spearman\n value: 73.72445893756499\n - type: manhattan_pearson\n value: 72.59932284620231\n - type: manhattan_spearman\n value: 73.68002490614583\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB CLSClusteringP2P\n revision: 4b6227591c6c1a73bc76b1055f3b7f3588e72476\n split: test\n type: C-MTEB/CLSClusteringP2P\n metrics:\n - type: v_measure\n value: 45.21317724305628\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB CLSClusteringS2S\n revision: e458b3f5414b62b7f9f83499ac1f5497ae2e869f\n split: test\n type: C-MTEB/CLSClusteringS2S\n metrics:\n - type: v_measure\n value: 42.49825170976724\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB CMedQAv1\n revision: 8d7f1e942507dac42dc58017c1a001c3717da7df\n split: test\n type: C-MTEB/CMedQAv1-reranking\n metrics:\n - type: map\n value: 88.15661686810597\n - type: mrr\n value: 90.11222222222223\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB CMedQAv2\n revision: 23d186750531a14a0357ca22cd92d712fd512ea0\n split: test\n type: C-MTEB/CMedQAv2-reranking\n metrics:\n - type: map\n value: 88.1204726064383\n - type: mrr\n value: 90.20142857142858\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB CmedqaRetrieval\n revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301\n split: dev\n type: C-MTEB/CmedqaRetrieval\n metrics:\n - type: map_at_1\n value: 27.224999999999998\n - type: map_at_10\n value: 40.169\n - type: map_at_100\n value: 42.0\n - type: map_at_1000\n value: 42.109\n - type: map_at_3\n value: 35.76\n - type: map_at_5\n value: 38.221\n - type: mrr_at_1\n value: 40.56\n - type: mrr_at_10\n value: 49.118\n - type: mrr_at_100\n value: 50.092999999999996\n - type: mrr_at_1000\n value: 50.133\n - type: mrr_at_3\n value: 46.507\n - type: mrr_at_5\n value: 47.973\n - type: ndcg_at_1\n value: 40.56\n - type: ndcg_at_10\n value: 46.972\n - type: ndcg_at_100\n value: 54.04\n - type: ndcg_at_1000\n value: 55.862\n - type: ndcg_at_3\n value: 41.36\n - type: ndcg_at_5\n value: 43.704\n - type: precision_at_1\n value: 40.56\n - type: precision_at_10\n value: 10.302999999999999\n - type: precision_at_100\n value: 1.606\n - type: precision_at_1000\n value: 0.184\n - type: precision_at_3\n value: 23.064\n - type: precision_at_5\n value: 16.764000000000003\n - type: recall_at_1\n value: 27.224999999999998\n - type: recall_at_10\n value: 58.05200000000001\n - type: recall_at_100\n value: 87.092\n - type: recall_at_1000\n value: 99.099\n - type: recall_at_3\n value: 41.373\n - type: recall_at_5\n value: 48.453\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB Cmnli\n revision: 41bc36f332156f7adc9e38f53777c959b2ae9766\n split: validation\n type: C-MTEB/CMNLI\n metrics:\n - type: cos_sim_accuracy\n value: 77.40228502705953\n - type: cos_sim_ap\n value: 86.22359172956327\n - type: cos_sim_f1\n value: 78.96328293736501\n - type: cos_sim_precision\n value: 73.36945615091311\n - type: cos_sim_recall\n value: 85.48047696983868\n - type: dot_accuracy\n value: 75.53818400481059\n - type: dot_ap\n value: 83.70164011305312\n - type: dot_f1\n value: 77.67298719348754\n - type: dot_precision\n value: 67.49482401656314\n - type: dot_recall\n value: 91.46598082768296\n - type: euclidean_accuracy\n value: 77.94347564642213\n - type: euclidean_ap\n value: 86.4652108728609\n - type: euclidean_f1\n value: 79.15555555555555\n - type: euclidean_precision\n value: 75.41816641964853\n - type: euclidean_recall\n value: 83.28267477203647\n - type: manhattan_accuracy\n value: 77.45039085989175\n - type: manhattan_ap\n value: 86.09986583900665\n - type: manhattan_f1\n value: 78.93669264438988\n - type: manhattan_precision\n value: 72.63261296660117\n - type: manhattan_recall\n value: 86.43909282207154\n - type: max_accuracy\n value: 77.94347564642213\n - type: max_ap\n value: 86.4652108728609\n - type: max_f1\n value: 79.15555555555555\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB CovidRetrieval\n revision: 1271c7809071a13532e05f25fb53511ffce77117\n split: dev\n type: C-MTEB/CovidRetrieval\n metrics:\n - type: map_at_1\n value: 69.336\n - type: map_at_10\n value: 77.16\n - type: map_at_100\n value: 77.47500000000001\n - type: map_at_1000\n value: 77.482\n - type: map_at_3\n value: 75.42999999999999\n - type: map_at_5\n value: 76.468\n - type: mrr_at_1\n value: 69.44200000000001\n - type: mrr_at_10\n value: 77.132\n - type: mrr_at_100\n value: 77.43299999999999\n - type: mrr_at_1000\n value: 77.44\n - type: mrr_at_3\n value: 75.395\n - type: mrr_at_5\n value: 76.459\n - type: ndcg_at_1\n value: 69.547\n - type: ndcg_at_10\n value: 80.794\n - type: ndcg_at_100\n value: 82.245\n - type: ndcg_at_1000\n value: 82.40899999999999\n - type: ndcg_at_3\n value: 77.303\n - type: ndcg_at_5\n value: 79.168\n - type: precision_at_1\n value: 69.547\n - type: precision_at_10\n value: 9.305\n - type: precision_at_100\n value: 0.9979999999999999\n - type: precision_at_1000\n value: 0.101\n - type: precision_at_3\n value: 27.749000000000002\n - type: precision_at_5\n value: 17.576\n - type: recall_at_1\n value: 69.336\n - type: recall_at_10\n value: 92.097\n - type: recall_at_100\n value: 98.736\n - type: recall_at_1000\n value: 100.0\n - type: recall_at_3\n value: 82.64\n - type: recall_at_5\n value: 87.144\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB DuRetrieval\n revision: a1a333e290fe30b10f3f56498e3a0d911a693ced\n split: dev\n type: C-MTEB/DuRetrieval\n metrics:\n - type: map_at_1\n value: 26.817999999999998\n - type: map_at_10\n value: 82.67\n - type: map_at_100\n value: 85.304\n - type: map_at_1000\n value: 85.334\n - type: map_at_3\n value: 57.336\n - type: map_at_5\n value: 72.474\n - type: mrr_at_1\n value: 91.45\n - type: mrr_at_10\n value: 94.272\n - type: mrr_at_100\n value: 94.318\n - type: mrr_at_1000\n value: 94.32000000000001\n - type: mrr_at_3\n value: 94.0\n - type: mrr_at_5\n value: 94.17699999999999\n - type: ndcg_at_1\n value: 91.45\n - type: ndcg_at_10\n value: 89.404\n - type: ndcg_at_100\n value: 91.724\n - type: ndcg_at_1000\n value: 91.973\n - type: ndcg_at_3\n value: 88.104\n - type: ndcg_at_5\n value: 87.25699999999999\n - type: precision_at_1\n value: 91.45\n - type: precision_at_10\n value: 42.585\n - type: precision_at_100\n value: 4.838\n - type: precision_at_1000\n value: 0.49\n - type: precision_at_3\n value: 78.8\n - type: precision_at_5\n value: 66.66\n - type: recall_at_1\n value: 26.817999999999998\n - type: recall_at_10\n value: 90.67\n - type: recall_at_100\n value: 98.36200000000001\n - type: recall_at_1000\n value: 99.583\n - type: recall_at_3\n value: 59.614999999999995\n - type: recall_at_5\n value: 77.05199999999999\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB EcomRetrieval\n revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9\n split: dev\n type: C-MTEB/EcomRetrieval\n metrics:\n - type: map_at_1\n value: 47.699999999999996\n - type: map_at_10\n value: 57.589999999999996\n - type: map_at_100\n value: 58.226\n - type: map_at_1000\n value: 58.251\n - type: map_at_3\n value: 55.233\n - type: map_at_5\n value: 56.633\n - type: mrr_at_1\n value: 47.699999999999996\n - type: mrr_at_10\n value: 57.589999999999996\n - type: mrr_at_100\n value: 58.226\n - type: mrr_at_1000\n value: 58.251\n - type: mrr_at_3\n value: 55.233\n - type: mrr_at_5\n value: 56.633\n - type: ndcg_at_1\n value: 47.699999999999996\n - type: ndcg_at_10\n value: 62.505\n - type: ndcg_at_100\n value: 65.517\n - type: ndcg_at_1000\n value: 66.19800000000001\n - type: ndcg_at_3\n value: 57.643\n - type: ndcg_at_5\n value: 60.181\n - type: precision_at_1\n value: 47.699999999999996\n - type: precision_at_10\n value: 7.8\n - type: precision_at_100\n value: 0.919\n - type: precision_at_1000\n value: 0.097\n - type: precision_at_3\n value: 21.532999999999998\n - type: precision_at_5\n value: 14.16\n - type: recall_at_1\n value: 47.699999999999996\n - type: recall_at_10\n value: 78.0\n - type: recall_at_100\n value: 91.9\n - type: recall_at_1000\n value: 97.3\n - type: recall_at_3\n value: 64.60000000000001\n - type: recall_at_5\n value: 70.8\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB IFlyTek\n revision: 421605374b29664c5fc098418fe20ada9bd55f8a\n split: validation\n type: C-MTEB/IFlyTek-classification\n metrics:\n - type: accuracy\n value: 44.84801846864178\n - type: f1\n value: 37.47347897956339\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB JDReview\n revision: b7c64bd89eb87f8ded463478346f76731f07bf8b\n split: test\n type: C-MTEB/JDReview-classification\n metrics:\n - type: accuracy\n value: 85.81613508442777\n - type: ap\n value: 52.68244615477374\n - type: f1\n value: 80.0445640948843\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB LCQMC\n revision: 17f9b096f80380fce5ed12a9be8be7784b337daf\n split: test\n type: C-MTEB/LCQMC\n metrics:\n - type: cos_sim_pearson\n value: 69.57786502217138\n - type: cos_sim_spearman\n value: 75.39106054489906\n - type: euclidean_pearson\n value: 73.72082954602402\n - type: euclidean_spearman\n value: 75.14421475913619\n - type: manhattan_pearson\n value: 73.62463076633642\n - type: manhattan_spearman\n value: 75.01301565104112\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB MMarcoReranking\n revision: None\n split: dev\n type: C-MTEB/Mmarco-reranking\n metrics:\n - type: map\n value: 29.143797057999134\n - type: mrr\n value: 28.08174603174603\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB MMarcoRetrieval\n revision: 539bbde593d947e2a124ba72651aafc09eb33fc2\n split: dev\n type: C-MTEB/MMarcoRetrieval\n metrics:\n - type: map_at_1\n value: 70.492\n - type: map_at_10\n value: 79.501\n - type: map_at_100\n value: 79.728\n - type: map_at_1000\n value: 79.735\n - type: map_at_3\n value: 77.77\n - type: map_at_5\n value: 78.851\n - type: mrr_at_1\n value: 72.822\n - type: mrr_at_10\n value: 80.001\n - type: mrr_at_100\n value: 80.19\n - type: mrr_at_1000\n value: 80.197\n - type: mrr_at_3\n value: 78.484\n - type: mrr_at_5\n value: 79.42099999999999\n - type: ndcg_at_1\n value: 72.822\n - type: ndcg_at_10\n value: 83.013\n - type: ndcg_at_100\n value: 84.013\n - type: ndcg_at_1000\n value: 84.20400000000001\n - type: ndcg_at_3\n value: 79.728\n - type: ndcg_at_5\n value: 81.542\n - type: precision_at_1\n value: 72.822\n - type: precision_at_10\n value: 9.917\n - type: precision_at_100\n value: 1.042\n - type: precision_at_1000\n value: 0.106\n - type: precision_at_3\n value: 29.847\n - type: precision_at_5\n value: 18.871\n - type: recall_at_1\n value: 70.492\n - type: recall_at_10\n value: 93.325\n - type: recall_at_100\n value: 97.822\n - type: recall_at_1000\n value: 99.319\n - type: recall_at_3\n value: 84.636\n - type: recall_at_5\n value: 88.93100000000001\n task:\n type: Retrieval\n - dataset:\n config: zh-CN\n name: MTEB MassiveIntentClassification (zh-CN)\n revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7\n split: test\n type: mteb/amazon_massive_intent\n metrics:\n - type: accuracy\n value: 76.88298587760592\n - type: f1\n value: 73.89001762017176\n task:\n type: Classification\n - dataset:\n config: zh-CN\n name: MTEB MassiveScenarioClassification (zh-CN)\n revision: 7d571f92784cd94a019292a1f45445077d0ef634\n split: test\n type: mteb/amazon_massive_scenario\n metrics:\n - type: accuracy\n value: 80.76328177538669\n - type: f1\n value: 80.24718532423358\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB MedicalRetrieval\n revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6\n split: dev\n type: C-MTEB/MedicalRetrieval\n metrics:\n - type: map_at_1\n value: 49.6\n - type: map_at_10\n value: 55.620999999999995\n - type: map_at_100\n value: 56.204\n - type: map_at_1000\n value: 56.251\n - type: map_at_3\n value: 54.132999999999996\n - type: map_at_5\n value: 54.933\n - type: mrr_at_1\n value: 49.7\n - type: mrr_at_10\n value: 55.67100000000001\n - type: mrr_at_100\n value: 56.254000000000005\n - type: mrr_at_1000\n value: 56.301\n - type: mrr_at_3\n value: 54.18300000000001\n - type: mrr_at_5\n value: 54.983000000000004\n - type: ndcg_at_1\n value: 49.6\n - type: ndcg_at_10\n value: 58.645\n - type: ndcg_at_100\n value: 61.789\n - type: ndcg_at_1000\n value: 63.219\n - type: ndcg_at_3\n value: 55.567\n - type: ndcg_at_5\n value: 57.008\n - type: precision_at_1\n value: 49.6\n - type: precision_at_10\n value: 6.819999999999999\n - type: precision_at_100\n value: 0.836\n - type: precision_at_1000\n value: 0.095\n - type: precision_at_3\n value: 19.900000000000002\n - type: precision_at_5\n value: 12.64\n - type: recall_at_1\n value: 49.6\n - type: recall_at_10\n value: 68.2\n - type: recall_at_100\n value: 83.6\n - type: recall_at_1000\n value: 95.3\n - type: recall_at_3\n value: 59.699999999999996\n - type: recall_at_5\n value: 63.2\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB MultilingualSentiment\n revision: 46958b007a63fdbf239b7672c25d0bea67b5ea1a\n split: validation\n type: C-MTEB/MultilingualSentiment-classification\n metrics:\n - type: accuracy\n value: 74.45666666666666\n - type: f1\n value: 74.32582402190089\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB Ocnli\n revision: 66e76a618a34d6d565d5538088562851e6daa7ec\n split: validation\n type: C-MTEB/OCNLI\n metrics:\n - type: cos_sim_accuracy\n value: 80.67135896047645\n - type: cos_sim_ap\n value: 87.60421240712051\n - type: cos_sim_f1\n value: 82.1304131408661\n - type: cos_sim_precision\n value: 77.68361581920904\n - type: cos_sim_recall\n value: 87.11721224920802\n - type: dot_accuracy\n value: 79.04710341093666\n - type: dot_ap\n value: 85.6370059719336\n - type: dot_f1\n value: 80.763723150358\n - type: dot_precision\n value: 73.69337979094077\n - type: dot_recall\n value: 89.33474128827878\n - type: euclidean_accuracy\n value: 81.05035192203573\n - type: euclidean_ap\n value: 87.7880240053663\n - type: euclidean_f1\n value: 82.50244379276637\n - type: euclidean_precision\n value: 76.7970882620564\n - type: euclidean_recall\n value: 89.1235480464625\n - type: manhattan_accuracy\n value: 80.61721710882512\n - type: manhattan_ap\n value: 87.43568120591175\n - type: manhattan_f1\n value: 81.89526184538653\n - type: manhattan_precision\n value: 77.5992438563327\n - type: manhattan_recall\n value: 86.6948257655755\n - type: max_accuracy\n value: 81.05035192203573\n - type: max_ap\n value: 87.7880240053663\n - type: max_f1\n value: 82.50244379276637\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB OnlineShopping\n revision: e610f2ebd179a8fda30ae534c3878750a96db120\n split: test\n type: C-MTEB/OnlineShopping-classification\n metrics:\n - type: accuracy\n value: 93.5\n - type: ap\n value: 91.31357903446782\n - type: f1\n value: 93.48088994006616\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB PAWSX\n revision: 9c6a90e430ac22b5779fb019a23e820b11a8b5e1\n split: test\n type: C-MTEB/PAWSX\n metrics:\n - type: cos_sim_pearson\n value: 36.93293453538077\n - type: cos_sim_spearman\n value: 42.45972506308574\n - type: euclidean_pearson\n value: 42.34945133152159\n - type: euclidean_spearman\n value: 42.331610303674644\n - type: manhattan_pearson\n value: 42.31455070249498\n - type: manhattan_spearman\n value: 42.19887982891834\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB QBQTC\n revision: 790b0510dc52b1553e8c49f3d2afb48c0e5c48b7\n split: test\n type: C-MTEB/QBQTC\n metrics:\n - type: cos_sim_pearson\n value: 33.683290790043785\n - type: cos_sim_spearman\n value: 35.149171171202994\n - type: euclidean_pearson\n value: 32.33806561267862\n - type: euclidean_spearman\n value: 34.483576387347966\n - type: manhattan_pearson\n value: 32.47629754599608\n - type: manhattan_spearman\n value: 34.66434471867615\n task:\n type: STS\n - dataset:\n config: zh\n name: MTEB STS22 (zh)\n revision: eea2b4fe26a775864c896887d910b76a8098ad3f\n split: test\n type: mteb/sts22-crosslingual-sts\n metrics:\n - type: cos_sim_pearson\n value: 66.46322760516104\n - type: cos_sim_spearman\n value: 67.398478319726\n - type: euclidean_pearson\n value: 64.7223480293625\n - type: euclidean_spearman\n value: 66.83118568812951\n - type: manhattan_pearson\n value: 64.88440039828305\n - type: manhattan_spearman\n value: 66.80429458952257\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB STSB\n revision: 0cde68302b3541bb8b3c340dc0644b0b745b3dc0\n split: test\n type: C-MTEB/STSB\n metrics:\n - type: cos_sim_pearson\n value: 79.08991383232105\n - type: cos_sim_spearman\n value: 79.39715677296854\n - type: euclidean_pearson\n value: 78.63201279320496\n - type: euclidean_spearman\n value: 79.40262660785731\n - type: manhattan_pearson\n value: 78.98138363146906\n - type: manhattan_spearman\n value: 79.79968413014194\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB T2Reranking\n revision: 76631901a18387f85eaa53e5450019b87ad58ef9\n split: dev\n type: C-MTEB/T2Reranking\n metrics:\n - type: map\n value: 67.43289278789972\n - type: mrr\n value: 77.53012460908535\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB T2Retrieval\n revision: 8731a845f1bf500a4f111cf1070785c793d10e64\n split: dev\n type: C-MTEB/T2Retrieval\n metrics:\n - type: map_at_1\n value: 27.733999999999998\n - type: map_at_10\n value: 78.24799999999999\n - type: map_at_100\n value: 81.765\n - type: map_at_1000\n value: 81.824\n - type: map_at_3\n value: 54.92\n - type: map_at_5\n value: 67.61399999999999\n - type: mrr_at_1\n value: 90.527\n - type: mrr_at_10\n value: 92.843\n - type: mrr_at_100\n value: 92.927\n - type: mrr_at_1000\n value: 92.93\n - type: mrr_at_3\n value: 92.45100000000001\n - type: mrr_at_5\n value: 92.693\n - type: ndcg_at_1\n value: 90.527\n - type: ndcg_at_10\n value: 85.466\n - type: ndcg_at_100\n value: 88.846\n - type: ndcg_at_1000\n value: 89.415\n - type: ndcg_at_3\n value: 86.768\n - type: ndcg_at_5\n value: 85.46000000000001\n - type: precision_at_1\n value: 90.527\n - type: precision_at_10\n value: 42.488\n - type: precision_at_100\n value: 5.024\n - type: precision_at_1000\n value: 0.516\n - type: precision_at_3\n value: 75.907\n - type: precision_at_5\n value: 63.727000000000004\n - type: recall_at_1\n value: 27.733999999999998\n - type: recall_at_10\n value: 84.346\n - type: recall_at_100\n value: 95.536\n - type: recall_at_1000\n value: 98.42999999999999\n - type: recall_at_3\n value: 56.455\n - type: recall_at_5\n value: 70.755\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB TNews\n revision: 317f262bf1e6126357bbe89e875451e4b0938fe4\n split: validation\n type: C-MTEB/TNews-classification\n metrics:\n - type: accuracy\n value: 49.952000000000005\n - type: f1\n value: 48.264617195258054\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB ThuNewsClusteringP2P\n revision: 5798586b105c0434e4f0fe5e767abe619442cf93\n split: test\n type: C-MTEB/ThuNewsClusteringP2P\n metrics:\n - type: v_measure\n value: 68.23769904483508\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB ThuNewsClusteringS2S\n revision: 8a8b2caeda43f39e13c4bc5bea0f8a667896e10d\n split: test\n type: C-MTEB/ThuNewsClusteringS2S\n metrics:\n - type: v_measure\n value: 62.50294403136556\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB VideoRetrieval\n revision: 58c2597a5943a2ba48f4668c3b90d796283c5639\n split: dev\n type: C-MTEB/VideoRetrieval\n metrics:\n - type: map_at_1\n value: 54.0\n - type: map_at_10\n value: 63.668\n - type: map_at_100\n value: 64.217\n - type: map_at_1000\n value: 64.23100000000001\n - type: map_at_3\n value: 61.7\n - type: map_at_5\n value: 62.870000000000005\n - type: mrr_at_1\n value: 54.0\n - type: mrr_at_10\n value: 63.668\n - type: mrr_at_100\n value: 64.217\n - type: mrr_at_1000\n value: 64.23100000000001\n - type: mrr_at_3\n value: 61.7\n - type: mrr_at_5\n value: 62.870000000000005\n - type: ndcg_at_1\n value: 54.0\n - type: ndcg_at_10\n value: 68.11399999999999\n - type: ndcg_at_100\n value: 70.723\n - type: ndcg_at_1000\n value: 71.123\n - type: ndcg_at_3\n value: 64.074\n - type: ndcg_at_5\n value: 66.178\n - type: precision_at_1\n value: 54.0\n - type: precision_at_10\n value: 8.200000000000001\n - type: precision_at_100\n value: 0.941\n - type: precision_at_1000\n value: 0.097\n - type: precision_at_3\n value: 23.633000000000003\n - type: precision_at_5\n value: 15.2\n - type: recall_at_1\n value: 54.0\n - type: recall_at_10\n value: 82.0\n - type: recall_at_100\n value: 94.1\n - type: recall_at_1000\n value: 97.3\n - type: recall_at_3\n value: 70.89999999999999\n - type: recall_at_5\n value: 76.0\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB Waimai\n revision: 339287def212450dcaa9df8c22bf93e9980c7023\n split: test\n type: C-MTEB/waimai-classification\n metrics:\n - type: accuracy\n value: 86.63000000000001\n - type: ap\n value: 69.99457882599567\n - type: f1\n value: 85.07735617998541\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB 8TagsClustering\n revision: None\n split: test\n type: PL-MTEB/8tags-clustering\n metrics:\n - type: v_measure\n value: 44.594104491193555\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB AllegroReviews\n revision: None\n split: test\n type: PL-MTEB/allegro-reviews\n metrics:\n - type: accuracy\n value: 63.97614314115309\n - type: f1\n value: 52.15634261679283\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB ArguAna-PL\n revision: 63fc86750af76253e8c760fc9e534bbf24d260a2\n split: test\n type: clarin-knext/arguana-pl\n metrics:\n - type: map_at_1\n value: 32.646\n - type: map_at_10\n value: 47.963\n - type: map_at_100\n value: 48.789\n - type: map_at_1000\n value: 48.797000000000004\n - type: map_at_3\n value: 43.196\n - type: map_at_5\n value: 46.016\n - type: mrr_at_1\n value: 33.073\n - type: mrr_at_10\n value: 48.126000000000005\n - type: mrr_at_100\n value: 48.946\n - type: mrr_at_1000\n value: 48.953\n - type: mrr_at_3\n value: 43.374\n - type: mrr_at_5\n value: 46.147\n - type: ndcg_at_1\n value: 32.646\n - type: ndcg_at_10\n value: 56.481\n - type: ndcg_at_100\n value: 59.922\n - type: ndcg_at_1000\n value: 60.07\n - type: ndcg_at_3\n value: 46.675\n - type: ndcg_at_5\n value: 51.76500000000001\n - type: precision_at_1\n value: 32.646\n - type: precision_at_10\n value: 8.371\n - type: precision_at_100\n value: 0.9860000000000001\n - type: precision_at_1000\n value: 0.1\n - type: precision_at_3\n value: 18.919\n - type: precision_at_5\n value: 13.825999999999999\n - type: recall_at_1\n value: 32.646\n - type: recall_at_10\n value: 83.71300000000001\n - type: recall_at_100\n value: 98.578\n - type: recall_at_1000\n value: 99.644\n - type: recall_at_3\n value: 56.757000000000005\n - type: recall_at_5\n value: 69.132\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB CBD\n revision: None\n split: test\n type: PL-MTEB/cbd\n metrics:\n - type: accuracy\n value: 68.56\n - type: ap\n value: 23.310493680488513\n - type: f1\n value: 58.85369533105693\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB CDSC-E\n revision: None\n split: test\n type: PL-MTEB/cdsce-pairclassification\n metrics:\n - type: cos_sim_accuracy\n value: 88.5\n - type: cos_sim_ap\n value: 72.42140924378361\n - type: cos_sim_f1\n value: 66.0919540229885\n - type: cos_sim_precision\n value: 72.78481012658227\n - type: cos_sim_recall\n value: 60.526315789473685\n - type: dot_accuracy\n value: 88.5\n - type: dot_ap\n value: 72.42140924378361\n - type: dot_f1\n value: 66.0919540229885\n - type: dot_precision\n value: 72.78481012658227\n - type: dot_recall\n value: 60.526315789473685\n - type: euclidean_accuracy\n value: 88.5\n - type: euclidean_ap\n value: 72.42140924378361\n - type: euclidean_f1\n value: 66.0919540229885\n - type: euclidean_precision\n value: 72.78481012658227\n - type: euclidean_recall\n value: 60.526315789473685\n - type: manhattan_accuracy\n value: 88.5\n - type: manhattan_ap\n value: 72.49745515311696\n - type: manhattan_f1\n value: 66.0968660968661\n - type: manhattan_precision\n value: 72.04968944099379\n - type: manhattan_recall\n value: 61.05263157894737\n - type: max_accuracy\n value: 88.5\n - type: max_ap\n value: 72.49745515311696\n - type: max_f1\n value: 66.0968660968661\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB CDSC-R\n revision: None\n split: test\n type: PL-MTEB/cdscr-sts\n metrics:\n - type: cos_sim_pearson\n value: 90.32269765590145\n - type: cos_sim_spearman\n value: 89.73666311491672\n - type: euclidean_pearson\n value: 88.2933868516544\n - type: euclidean_spearman\n value: 89.73666311491672\n - type: manhattan_pearson\n value: 88.33474590219448\n - type: manhattan_spearman\n value: 89.8548364866583\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB DBPedia-PL\n revision: 76afe41d9af165cc40999fcaa92312b8b012064a\n split: test\n type: clarin-knext/dbpedia-pl\n metrics:\n - type: map_at_1\n value: 7.632999999999999\n - type: map_at_10\n value: 16.426\n - type: map_at_100\n value: 22.651\n - type: map_at_1000\n value: 24.372\n - type: map_at_3\n value: 11.706\n - type: map_at_5\n value: 13.529\n - type: mrr_at_1\n value: 60.75000000000001\n - type: mrr_at_10\n value: 68.613\n - type: mrr_at_100\n value: 69.001\n - type: mrr_at_1000\n value: 69.021\n - type: mrr_at_3\n value: 67.0\n - type: mrr_at_5\n value: 67.925\n - type: ndcg_at_1\n value: 49.875\n - type: ndcg_at_10\n value: 36.978\n - type: ndcg_at_100\n value: 40.031\n - type: ndcg_at_1000\n value: 47.566\n - type: ndcg_at_3\n value: 41.148\n - type: ndcg_at_5\n value: 38.702\n - type: precision_at_1\n value: 60.75000000000001\n - type: precision_at_10\n value: 29.7\n - type: precision_at_100\n value: 9.278\n - type: precision_at_1000\n value: 2.099\n - type: precision_at_3\n value: 44.0\n - type: precision_at_5\n value: 37.6\n - type: recall_at_1\n value: 7.632999999999999\n - type: recall_at_10\n value: 22.040000000000003\n - type: recall_at_100\n value: 44.024\n - type: recall_at_1000\n value: 67.848\n - type: recall_at_3\n value: 13.093\n - type: recall_at_5\n value: 15.973\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB FiQA-PL\n revision: 2e535829717f8bf9dc829b7f911cc5bbd4e6608e\n split: test\n type: clarin-knext/fiqa-pl\n metrics:\n - type: map_at_1\n value: 15.473\n - type: map_at_10\n value: 24.579\n - type: map_at_100\n value: 26.387\n - type: map_at_1000\n value: 26.57\n - type: map_at_3\n value: 21.278\n - type: map_at_5\n value: 23.179\n - type: mrr_at_1\n value: 30.709999999999997\n - type: mrr_at_10\n value: 38.994\n - type: mrr_at_100\n value: 39.993\n - type: mrr_at_1000\n value: 40.044999999999995\n - type: mrr_at_3\n value: 36.342999999999996\n - type: mrr_at_5\n value: 37.846999999999994\n - type: ndcg_at_1\n value: 30.709999999999997\n - type: ndcg_at_10\n value: 31.608999999999998\n - type: ndcg_at_100\n value: 38.807\n - type: ndcg_at_1000\n value: 42.208\n - type: ndcg_at_3\n value: 28.086\n - type: ndcg_at_5\n value: 29.323\n - type: precision_at_1\n value: 30.709999999999997\n - type: precision_at_10\n value: 8.688\n - type: precision_at_100\n value: 1.608\n - type: precision_at_1000\n value: 0.22100000000000003\n - type: precision_at_3\n value: 18.724\n - type: precision_at_5\n value: 13.950999999999999\n - type: recall_at_1\n value: 15.473\n - type: recall_at_10\n value: 38.361000000000004\n - type: recall_at_100\n value: 65.2\n - type: recall_at_1000\n value: 85.789\n - type: recall_at_3\n value: 25.401\n - type: recall_at_5\n value: 30.875999999999998\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB HotpotQA-PL\n revision: a0bd479ac97b4ccb5bd6ce320c415d0bb4beb907\n split: test\n type: clarin-knext/hotpotqa-pl\n metrics:\n - type: map_at_1\n value: 38.096000000000004\n - type: map_at_10\n value: 51.44499999999999\n - type: map_at_100\n value: 52.325\n - type: map_at_1000\n value: 52.397000000000006\n - type: map_at_3\n value: 48.626999999999995\n - type: map_at_5\n value: 50.342\n - type: mrr_at_1\n value: 76.19200000000001\n - type: mrr_at_10\n value: 81.191\n - type: mrr_at_100\n value: 81.431\n - type: mrr_at_1000\n value: 81.443\n - type: mrr_at_3\n value: 80.30199999999999\n - type: mrr_at_5\n value: 80.85900000000001\n - type: ndcg_at_1\n value: 76.19200000000001\n - type: ndcg_at_10\n value: 60.9\n - type: ndcg_at_100\n value: 64.14699999999999\n - type: ndcg_at_1000\n value: 65.647\n - type: ndcg_at_3\n value: 56.818000000000005\n - type: ndcg_at_5\n value: 59.019999999999996\n - type: precision_at_1\n value: 76.19200000000001\n - type: precision_at_10\n value: 12.203\n - type: precision_at_100\n value: 1.478\n - type: precision_at_1000\n value: 0.168\n - type: precision_at_3\n value: 34.616\n - type: precision_at_5\n value: 22.515\n - type: recall_at_1\n value: 38.096000000000004\n - type: recall_at_10\n value: 61.013\n - type: recall_at_100\n value: 73.90299999999999\n - type: recall_at_1000\n value: 83.91\n - type: recall_at_3\n value: 51.92400000000001\n - type: recall_at_5\n value: 56.286\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB MSMARCO-PL\n revision: 8634c07806d5cce3a6138e260e59b81760a0a640\n split: test\n type: clarin-knext/msmarco-pl\n metrics:\n - type: map_at_1\n value: 1.548\n - type: map_at_10\n value: 11.049000000000001\n - type: map_at_100\n value: 28.874\n - type: map_at_1000\n value: 34.931\n - type: map_at_3\n value: 4.162\n - type: map_at_5\n value: 6.396\n - type: mrr_at_1\n value: 90.69800000000001\n - type: mrr_at_10\n value: 92.093\n - type: mrr_at_100\n value: 92.345\n - type: mrr_at_1000\n value: 92.345\n - type: mrr_at_3\n value: 91.86\n - type: mrr_at_5\n value: 91.86\n - type: ndcg_at_1\n value: 74.031\n - type: ndcg_at_10\n value: 63.978\n - type: ndcg_at_100\n value: 53.101\n - type: ndcg_at_1000\n value: 60.675999999999995\n - type: ndcg_at_3\n value: 71.421\n - type: ndcg_at_5\n value: 68.098\n - type: precision_at_1\n value: 90.69800000000001\n - type: precision_at_10\n value: 71.86\n - type: precision_at_100\n value: 31.395\n - type: precision_at_1000\n value: 5.981\n - type: precision_at_3\n value: 84.49600000000001\n - type: precision_at_5\n value: 79.07\n - type: recall_at_1\n value: 1.548\n - type: recall_at_10\n value: 12.149000000000001\n - type: recall_at_100\n value: 40.794999999999995\n - type: recall_at_1000\n value: 67.974\n - type: recall_at_3\n value: 4.244\n - type: recall_at_5\n value: 6.608\n task:\n type: Retrieval\n - dataset:\n config: pl\n name: MTEB MassiveIntentClassification (pl)\n revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7\n split: test\n type: mteb/amazon_massive_intent\n metrics:\n - type: accuracy\n value: 73.55413584398119\n - type: f1\n value: 69.65610882318181\n task:\n type: Classification\n - dataset:\n config: pl\n name: MTEB MassiveScenarioClassification (pl)\n revision: 7d571f92784cd94a019292a1f45445077d0ef634\n split: test\n type: mteb/amazon_massive_scenario\n metrics:\n - type: accuracy\n value: 76.37188971082716\n - type: f1\n value: 75.64847309941361\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB NFCorpus-PL\n revision: 9a6f9567fda928260afed2de480d79c98bf0bec0\n split: test\n type: clarin-knext/nfcorpus-pl\n metrics:\n - type: map_at_1\n value: 4.919\n - type: map_at_10\n value: 10.834000000000001\n - type: map_at_100\n value: 13.38\n - type: map_at_1000\n value: 14.581\n - type: map_at_3\n value: 8.198\n - type: map_at_5\n value: 9.428\n - type: mrr_at_1\n value: 41.176\n - type: mrr_at_10\n value: 50.083\n - type: mrr_at_100\n value: 50.559\n - type: mrr_at_1000\n value: 50.604000000000006\n - type: mrr_at_3\n value: 47.936\n - type: mrr_at_5\n value: 49.407000000000004\n - type: ndcg_at_1\n value: 39.628\n - type: ndcg_at_10\n value: 30.098000000000003\n - type: ndcg_at_100\n value: 27.061\n - type: ndcg_at_1000\n value: 35.94\n - type: ndcg_at_3\n value: 35.135\n - type: ndcg_at_5\n value: 33.335\n - type: precision_at_1\n value: 41.176\n - type: precision_at_10\n value: 22.259999999999998\n - type: precision_at_100\n value: 6.712\n - type: precision_at_1000\n value: 1.9060000000000001\n - type: precision_at_3\n value: 33.23\n - type: precision_at_5\n value: 29.04\n - type: recall_at_1\n value: 4.919\n - type: recall_at_10\n value: 14.196\n - type: recall_at_100\n value: 26.948\n - type: recall_at_1000\n value: 59.211000000000006\n - type: recall_at_3\n value: 9.44\n - type: recall_at_5\n value: 11.569\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB NQ-PL\n revision: f171245712cf85dd4700b06bef18001578d0ca8d\n split: test\n type: clarin-knext/nq-pl\n metrics:\n - type: map_at_1\n value: 25.35\n - type: map_at_10\n value: 37.884\n - type: map_at_100\n value: 38.955\n - type: map_at_1000\n value: 39.007999999999996\n - type: map_at_3\n value: 34.239999999999995\n - type: map_at_5\n value: 36.398\n - type: mrr_at_1\n value: 28.737000000000002\n - type: mrr_at_10\n value: 39.973\n - type: mrr_at_100\n value: 40.844\n - type: mrr_at_1000\n value: 40.885\n - type: mrr_at_3\n value: 36.901\n - type: mrr_at_5\n value: 38.721\n - type: ndcg_at_1\n value: 28.708\n - type: ndcg_at_10\n value: 44.204\n - type: ndcg_at_100\n value: 48.978\n - type: ndcg_at_1000\n value: 50.33\n - type: ndcg_at_3\n value: 37.36\n - type: ndcg_at_5\n value: 40.912\n - type: precision_at_1\n value: 28.708\n - type: precision_at_10\n value: 7.367\n - type: precision_at_100\n value: 1.0030000000000001\n - type: precision_at_1000\n value: 0.11299999999999999\n - type: precision_at_3\n value: 17.034\n - type: precision_at_5\n value: 12.293999999999999\n - type: recall_at_1\n value: 25.35\n - type: recall_at_10\n value: 61.411\n - type: recall_at_100\n value: 82.599\n - type: recall_at_1000\n value: 92.903\n - type: recall_at_3\n value: 43.728\n - type: recall_at_5\n value: 51.854\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB PAC\n revision: None\n split: test\n type: laugustyniak/abusive-clauses-pl\n metrics:\n - type: accuracy\n value: 69.04141326382856\n - type: ap\n value: 77.49422763833996\n - type: f1\n value: 66.73472657783407\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB PPC\n revision: None\n split: test\n type: PL-MTEB/ppc-pairclassification\n metrics:\n - type: cos_sim_accuracy\n value: 81.0\n - type: cos_sim_ap\n value: 91.47194213011349\n - type: cos_sim_f1\n value: 84.73767885532592\n - type: cos_sim_precision\n value: 81.49847094801224\n - type: cos_sim_recall\n value: 88.24503311258279\n - type: dot_accuracy\n value: 81.0\n - type: dot_ap\n value: 91.47194213011349\n - type: dot_f1\n value: 84.73767885532592\n - type: dot_precision\n value: 81.49847094801224\n - type: dot_recall\n value: 88.24503311258279\n - type: euclidean_accuracy\n value: 81.0\n - type: euclidean_ap\n value: 91.47194213011349\n - type: euclidean_f1\n value: 84.73767885532592\n - type: euclidean_precision\n value: 81.49847094801224\n - type: euclidean_recall\n value: 88.24503311258279\n - type: manhattan_accuracy\n value: 81.0\n - type: manhattan_ap\n value: 91.46464475050571\n - type: manhattan_f1\n value: 84.48687350835321\n - type: manhattan_precision\n value: 81.31699846860643\n - type: manhattan_recall\n value: 87.91390728476821\n - type: max_accuracy\n value: 81.0\n - type: max_ap\n value: 91.47194213011349\n - type: max_f1\n value: 84.73767885532592\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB PSC\n revision: None\n split: test\n type: PL-MTEB/psc-pairclassification\n metrics:\n - type: cos_sim_accuracy\n value: 97.6808905380334\n - type: cos_sim_ap\n value: 99.27948611836348\n - type: cos_sim_f1\n value: 96.15975422427034\n - type: cos_sim_precision\n value: 96.90402476780186\n - type: cos_sim_recall\n value: 95.42682926829268\n - type: dot_accuracy\n value: 97.6808905380334\n - type: dot_ap\n value: 99.2794861183635\n - type: dot_f1\n value: 96.15975422427034\n - type: dot_precision\n value: 96.90402476780186\n - type: dot_recall\n value: 95.42682926829268\n - type: euclidean_accuracy\n value: 97.6808905380334\n - type: euclidean_ap\n value: 99.2794861183635\n - type: euclidean_f1\n value: 96.15975422427034\n - type: euclidean_precision\n value: 96.90402476780186\n - type: euclidean_recall\n value: 95.42682926829268\n - type: manhattan_accuracy\n value: 97.6808905380334\n - type: manhattan_ap\n value: 99.28715055268721\n - type: manhattan_f1\n value: 96.14791987673343\n - type: manhattan_precision\n value: 97.19626168224299\n - type: manhattan_recall\n value: 95.1219512195122\n - type: max_accuracy\n value: 97.6808905380334\n - type: max_ap\n value: 99.28715055268721\n - type: max_f1\n value: 96.15975422427034\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB PolEmo2.0-IN\n revision: None\n split: test\n type: PL-MTEB/polemo2_in\n metrics:\n - type: accuracy\n value: 86.16343490304708\n - type: f1\n value: 83.3442579486744\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB PolEmo2.0-OUT\n revision: None\n split: test\n type: PL-MTEB/polemo2_out\n metrics:\n - type: accuracy\n value: 68.40080971659918\n - type: f1\n value: 53.13720751142237\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB Quora-PL\n revision: 0be27e93455051e531182b85e85e425aba12e9d4\n split: test\n type: clarin-knext/quora-pl\n metrics:\n - type: map_at_1\n value: 63.322\n - type: map_at_10\n value: 76.847\n - type: map_at_100\n value: 77.616\n - type: map_at_1000\n value: 77.644\n - type: map_at_3\n value: 73.624\n - type: map_at_5\n value: 75.603\n - type: mrr_at_1\n value: 72.88\n - type: mrr_at_10\n value: 80.376\n - type: mrr_at_100\n value: 80.604\n - type: mrr_at_1000\n value: 80.61\n - type: mrr_at_3\n value: 78.92\n - type: mrr_at_5\n value: 79.869\n - type: ndcg_at_1\n value: 72.89999999999999\n - type: ndcg_at_10\n value: 81.43\n - type: ndcg_at_100\n value: 83.394\n - type: ndcg_at_1000\n value: 83.685\n - type: ndcg_at_3\n value: 77.62599999999999\n - type: ndcg_at_5\n value: 79.656\n - type: precision_at_1\n value: 72.89999999999999\n - type: precision_at_10\n value: 12.548\n - type: precision_at_100\n value: 1.4869999999999999\n - type: precision_at_1000\n value: 0.155\n - type: precision_at_3\n value: 34.027\n - type: precision_at_5\n value: 22.654\n - type: recall_at_1\n value: 63.322\n - type: recall_at_10\n value: 90.664\n - type: recall_at_100\n value: 97.974\n - type: recall_at_1000\n value: 99.636\n - type: recall_at_3\n value: 80.067\n - type: recall_at_5\n value: 85.526\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB SCIDOCS-PL\n revision: 45452b03f05560207ef19149545f168e596c9337\n split: test\n type: clarin-knext/scidocs-pl\n metrics:\n - type: map_at_1\n value: 3.95\n - type: map_at_10\n value: 9.658999999999999\n - type: map_at_100\n value: 11.384\n - type: map_at_1000\n value: 11.677\n - type: map_at_3\n value: 7.055\n - type: map_at_5\n value: 8.244\n - type: mrr_at_1\n value: 19.5\n - type: mrr_at_10\n value: 28.777\n - type: mrr_at_100\n value: 29.936\n - type: mrr_at_1000\n value: 30.009999999999998\n - type: mrr_at_3\n value: 25.55\n - type: mrr_at_5\n value: 27.284999999999997\n - type: ndcg_at_1\n value: 19.5\n - type: ndcg_at_10\n value: 16.589000000000002\n - type: ndcg_at_100\n value: 23.879\n - type: ndcg_at_1000\n value: 29.279\n - type: ndcg_at_3\n value: 15.719\n - type: ndcg_at_5\n value: 13.572000000000001\n - type: precision_at_1\n value: 19.5\n - type: precision_at_10\n value: 8.62\n - type: precision_at_100\n value: 1.924\n - type: precision_at_1000\n value: 0.322\n - type: precision_at_3\n value: 14.6\n - type: precision_at_5\n value: 11.78\n - type: recall_at_1\n value: 3.95\n - type: recall_at_10\n value: 17.477999999999998\n - type: recall_at_100\n value: 38.99\n - type: recall_at_1000\n value: 65.417\n - type: recall_at_3\n value: 8.883000000000001\n - type: recall_at_5\n value: 11.933\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB SICK-E-PL\n revision: None\n split: test\n type: PL-MTEB/sicke-pl-pairclassification\n metrics:\n - type: cos_sim_accuracy\n value: 83.48960456583775\n - type: cos_sim_ap\n value: 76.31522115825375\n - type: cos_sim_f1\n value: 70.35573122529645\n - type: cos_sim_precision\n value: 70.9934735315446\n - type: cos_sim_recall\n value: 69.72934472934473\n - type: dot_accuracy\n value: 83.48960456583775\n - type: dot_ap\n value: 76.31522115825373\n - type: dot_f1\n value: 70.35573122529645\n - type: dot_precision\n value: 70.9934735315446\n - type: dot_recall\n value: 69.72934472934473\n - type: euclidean_accuracy\n value: 83.48960456583775\n - type: euclidean_ap\n value: 76.31522115825373\n - type: euclidean_f1\n value: 70.35573122529645\n - type: euclidean_precision\n value: 70.9934735315446\n - type: euclidean_recall\n value: 69.72934472934473\n - type: manhattan_accuracy\n value: 83.46922136159804\n - type: manhattan_ap\n value: 76.18474601388084\n - type: manhattan_f1\n value: 70.34779490856937\n - type: manhattan_precision\n value: 70.83032490974729\n - type: manhattan_recall\n value: 69.87179487179486\n - type: max_accuracy\n value: 83.48960456583775\n - type: max_ap\n value: 76.31522115825375\n - type: max_f1\n value: 70.35573122529645\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB SICK-R-PL\n revision: None\n split: test\n type: PL-MTEB/sickr-pl-sts\n metrics:\n - type: cos_sim_pearson\n value: 77.95374883876302\n - type: cos_sim_spearman\n value: 73.77630219171942\n - type: euclidean_pearson\n value: 75.81927069594934\n - type: euclidean_spearman\n value: 73.7763211303831\n - type: manhattan_pearson\n value: 76.03126859057528\n - type: manhattan_spearman\n value: 73.96528138013369\n task:\n type: STS\n - dataset:\n config: pl\n name: MTEB STS22 (pl)\n revision: eea2b4fe26a775864c896887d910b76a8098ad3f\n split: test\n type: mteb/sts22-crosslingual-sts\n metrics:\n - type: cos_sim_pearson\n value: 37.388282764841826\n - type: cos_sim_spearman\n value: 40.83477184710897\n - type: euclidean_pearson\n value: 26.754737044177805\n - type: euclidean_spearman\n value: 40.83477184710897\n - type: manhattan_pearson\n value: 26.760453110872458\n - type: manhattan_spearman\n value: 41.034477441383856\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB SciFact-PL\n revision: 47932a35f045ef8ed01ba82bf9ff67f6e109207e\n split: test\n type: clarin-knext/scifact-pl\n metrics:\n - type: map_at_1\n value: 49.15\n - type: map_at_10\n value: 61.690999999999995\n - type: map_at_100\n value: 62.348000000000006\n - type: map_at_1000\n value: 62.38\n - type: map_at_3\n value: 58.824\n - type: map_at_5\n value: 60.662000000000006\n - type: mrr_at_1\n value: 51.333\n - type: mrr_at_10\n value: 62.731\n - type: mrr_at_100\n value: 63.245\n - type: mrr_at_1000\n value: 63.275000000000006\n - type: mrr_at_3\n value: 60.667\n - type: mrr_at_5\n value: 61.93300000000001\n - type: ndcg_at_1\n value: 51.333\n - type: ndcg_at_10\n value: 67.168\n - type: ndcg_at_100\n value: 69.833\n - type: ndcg_at_1000\n value: 70.56700000000001\n - type: ndcg_at_3\n value: 62.40599999999999\n - type: ndcg_at_5\n value: 65.029\n - type: precision_at_1\n value: 51.333\n - type: precision_at_10\n value: 9.333\n - type: precision_at_100\n value: 1.0699999999999998\n - type: precision_at_1000\n value: 0.11299999999999999\n - type: precision_at_3\n value: 25.333\n - type: precision_at_5\n value: 17.067\n - type: recall_at_1\n value: 49.15\n - type: recall_at_10\n value: 82.533\n - type: recall_at_100\n value: 94.167\n - type: recall_at_1000\n value: 99.667\n - type: recall_at_3\n value: 69.917\n - type: recall_at_5\n value: 76.356\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB TRECCOVID-PL\n revision: 81bcb408f33366c2a20ac54adafad1ae7e877fdd\n split: test\n type: clarin-knext/trec-covid-pl\n metrics:\n - type: map_at_1\n value: 0.261\n - type: map_at_10\n value: 2.1260000000000003\n - type: map_at_100\n value: 12.171999999999999\n - type: map_at_1000\n value: 26.884999999999998\n - type: map_at_3\n value: 0.695\n - type: map_at_5\n value: 1.134\n - type: mrr_at_1\n value: 96.0\n - type: mrr_at_10\n value: 96.952\n - type: mrr_at_100\n value: 96.952\n - type: mrr_at_1000\n value: 96.952\n - type: mrr_at_3\n value: 96.667\n - type: mrr_at_5\n value: 96.667\n - type: ndcg_at_1\n value: 92.0\n - type: ndcg_at_10\n value: 81.193\n - type: ndcg_at_100\n value: 61.129\n - type: ndcg_at_1000\n value: 51.157\n - type: ndcg_at_3\n value: 85.693\n - type: ndcg_at_5\n value: 84.129\n - type: precision_at_1\n value: 96.0\n - type: precision_at_10\n value: 85.39999999999999\n - type: precision_at_100\n value: 62.03999999999999\n - type: precision_at_1000\n value: 22.224\n - type: precision_at_3\n value: 88.0\n - type: precision_at_5\n value: 88.0\n - type: recall_at_1\n value: 0.261\n - type: recall_at_10\n value: 2.262\n - type: recall_at_100\n value: 14.981\n - type: recall_at_1000\n value: 46.837\n - type: recall_at_3\n value: 0.703\n - type: recall_at_5\n value: 1.172\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB AlloProfClusteringP2P\n revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b\n split: test\n type: lyon-nlp/alloprof\n metrics:\n - type: v_measure\n value: 70.55290063940157\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB AlloProfClusteringS2S\n revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b\n split: test\n type: lyon-nlp/alloprof\n metrics:\n - type: v_measure\n value: 55.41500719337263\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB AlloprofReranking\n revision: 666fdacebe0291776e86f29345663dfaf80a0db9\n split: test\n type: lyon-nlp/mteb-fr-reranking-alloprof-s2p\n metrics:\n - type: map\n value: 73.48697375332002\n - type: mrr\n value: 75.01836585523822\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB AlloprofRetrieval\n revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b\n split: test\n type: lyon-nlp/alloprof\n metrics:\n - type: map_at_1\n value: 38.454\n - type: map_at_10\n value: 51.605000000000004\n - type: map_at_100\n value: 52.653000000000006\n - type: map_at_1000\n value: 52.697\n - type: map_at_3\n value: 48.304\n - type: map_at_5\n value: 50.073\n - type: mrr_at_1\n value: 43.307\n - type: mrr_at_10\n value: 54.400000000000006\n - type: mrr_at_100\n value: 55.147999999999996\n - type: mrr_at_1000\n value: 55.174\n - type: mrr_at_3\n value: 51.77\n - type: mrr_at_5\n value: 53.166999999999994\n - type: ndcg_at_1\n value: 43.307\n - type: ndcg_at_10\n value: 57.891000000000005\n - type: ndcg_at_100\n value: 62.161\n - type: ndcg_at_1000\n value: 63.083\n - type: ndcg_at_3\n value: 51.851\n - type: ndcg_at_5\n value: 54.605000000000004\n - type: precision_at_1\n value: 43.307\n - type: precision_at_10\n value: 9.033\n - type: precision_at_100\n value: 1.172\n - type: precision_at_1000\n value: 0.127\n - type: precision_at_3\n value: 22.798\n - type: precision_at_5\n value: 15.492\n - type: recall_at_1\n value: 38.454\n - type: recall_at_10\n value: 74.166\n - type: recall_at_100\n value: 92.43599999999999\n - type: recall_at_1000\n value: 99.071\n - type: recall_at_3\n value: 58.087\n - type: recall_at_5\n value: 64.568\n task:\n type: Retrieval\n - dataset:\n config: fr\n name: MTEB AmazonReviewsClassification (fr)\n revision: 1399c76144fd37290681b995c656ef9b2e06e26d\n split: test\n type: mteb/amazon_reviews_multi\n metrics:\n - type: accuracy\n value: 53.474\n - type: f1\n value: 50.38275392350236\n task:\n type: Classification\n - dataset:\n config: default\n name: MTEB BSARDRetrieval\n revision: 5effa1b9b5fa3b0f9e12523e6e43e5f86a6e6d59\n split: test\n type: maastrichtlawtech/bsard\n metrics:\n - type: map_at_1\n value: 2.252\n - type: map_at_10\n value: 4.661\n - type: map_at_100\n value: 5.271\n - type: map_at_1000\n value: 5.3629999999999995\n - type: map_at_3\n value: 3.604\n - type: map_at_5\n value: 4.3020000000000005\n - type: mrr_at_1\n value: 2.252\n - type: mrr_at_10\n value: 4.661\n - type: mrr_at_100\n value: 5.271\n - type: mrr_at_1000\n value: 5.3629999999999995\n - type: mrr_at_3\n value: 3.604\n - type: mrr_at_5\n value: 4.3020000000000005\n - type: ndcg_at_1\n value: 2.252\n - type: ndcg_at_10\n value: 6.3020000000000005\n - type: ndcg_at_100\n value: 10.342\n - type: ndcg_at_1000\n value: 13.475999999999999\n - type: ndcg_at_3\n value: 4.0649999999999995\n - type: ndcg_at_5\n value: 5.344\n - type: precision_at_1\n value: 2.252\n - type: precision_at_10\n value: 1.171\n - type: precision_at_100\n value: 0.333\n - type: precision_at_1000\n value: 0.059000000000000004\n - type: precision_at_3\n value: 1.802\n - type: precision_at_5\n value: 1.712\n - type: recall_at_1\n value: 2.252\n - type: recall_at_10\n value: 11.712\n - type: recall_at_100\n value: 33.333\n - type: recall_at_1000\n value: 59.458999999999996\n - type: recall_at_3\n value: 5.405\n - type: recall_at_5\n value: 8.559\n task:\n type: Retrieval\n - dataset:\n config: default\n name: MTEB HALClusteringS2S\n revision: e06ebbbb123f8144bef1a5d18796f3dec9ae2915\n split: test\n type: lyon-nlp/clustering-hal-s2s\n metrics:\n - type: v_measure\n value: 28.301882091023288\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB MLSUMClusteringP2P\n revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7\n split: test\n type: mlsum\n metrics:\n - type: v_measure\n value: 45.26992995191701\n task:\n type: Clustering\n - dataset:\n config: default\n name: MTEB MLSUMClusteringS2S\n revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7\n split: test\n type: mlsum\n metrics:\n - type: v_measure\n value: 42.773174876871145\n task:\n type: Clustering\n - dataset:\n config: fr\n name: MTEB MTOPDomainClassification (fr)\n revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf\n split: test\n type: mteb/mtop_domain\n metrics:\n - type: accuracy\n value: 93.47635452552458\n - type: f1\n value: 93.19922617577213\n task:\n type: Classification\n - dataset:\n config: fr\n name: MTEB MTOPIntentClassification (fr)\n revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba\n split: test\n type: mteb/mtop_intent\n metrics:\n - type: accuracy\n value: 80.2317569683683\n - type: f1\n value: 56.18060418621901\n task:\n type: Classification\n - dataset:\n config: fra\n name: MTEB MasakhaNEWSClassification (fra)\n revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60\n split: test\n type: masakhane/masakhanews\n metrics:\n - type: accuracy\n value: 85.18957345971565\n - type: f1\n value: 80.829981537394\n task:\n type: Classification\n - dataset:\n config: fra\n name: MTEB MasakhaNEWSClusteringP2P (fra)\n revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60\n split: test\n type: masakhane/masakhanews\n metrics:\n - type: v_measure\n value: 71.04138999801822\n task:\n type: Clustering\n - dataset:\n config: fra\n name: MTEB MasakhaNEWSClusteringS2S (fra)\n revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60\n split: test\n type: masakhane/masakhanews\n metrics:\n - type: v_measure\n value: 71.7056263158008\n task:\n type: Clustering\n - dataset:\n config: fr\n name: MTEB MassiveIntentClassification (fr)\n revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7\n split: test\n type: mteb/amazon_massive_intent\n metrics:\n - type: accuracy\n value: 76.65097511768661\n - type: f1\n value: 73.82441070598712\n task:\n type: Classification\n - dataset:\n config: fr\n name: MTEB MassiveScenarioClassification (fr)\n revision: 7d571f92784cd94a019292a1f45445077d0ef634\n split: test\n type: mteb/amazon_massive_scenario\n metrics:\n - type: accuracy\n value: 79.09885675857431\n - type: f1\n value: 78.28407777434224\n task:\n type: Classification\n - dataset:\n config: fr\n name: MTEB MintakaRetrieval (fr)\n revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e\n split: test\n type: jinaai/mintakaqa\n metrics:\n - type: map_at_1\n value: 25.307000000000002\n - type: map_at_10\n value: 36.723\n - type: map_at_100\n value: 37.713\n - type: map_at_1000\n value: 37.769000000000005\n - type: map_at_3\n value: 33.77\n - type: map_at_5\n value: 35.463\n - type: mrr_at_1\n value: 25.307000000000002\n - type: mrr_at_10\n value: 36.723\n - type: mrr_at_100\n value: 37.713\n - type: mrr_at_1000\n value: 37.769000000000005\n - type: mrr_at_3\n value: 33.77\n - type: mrr_at_5\n value: 35.463\n - type: ndcg_at_1\n value: 25.307000000000002\n - type: ndcg_at_10\n value: 42.559999999999995\n - type: ndcg_at_100\n value: 47.457\n - type: ndcg_at_1000\n value: 49.162\n - type: ndcg_at_3\n value: 36.461\n - type: ndcg_at_5\n value: 39.504\n - type: precision_at_1\n value: 25.307000000000002\n - type: precision_at_10\n value: 6.106\n - type: precision_at_100\n value: 0.8420000000000001\n - type: precision_at_1000\n value: 0.098\n - type: precision_at_3\n value: 14.741999999999999\n - type: precision_at_5\n value: 10.319\n - type: recall_at_1\n value: 25.307000000000002\n - type: recall_at_10\n value: 61.056999999999995\n - type: recall_at_100\n value: 84.152\n - type: recall_at_1000\n value: 98.03399999999999\n - type: recall_at_3\n value: 44.226\n - type: recall_at_5\n value: 51.597\n task:\n type: Retrieval\n - dataset:\n config: fr\n name: MTEB OpusparcusPC (fr)\n revision: 9e9b1f8ef51616073f47f306f7f47dd91663f86a\n split: test\n type: GEM/opusparcus\n metrics:\n - type: cos_sim_accuracy\n value: 99.90069513406156\n - type: cos_sim_ap\n value: 100.0\n - type: cos_sim_f1\n value: 99.95032290114257\n - type: cos_sim_precision\n value: 100.0\n - type: cos_sim_recall\n value: 99.90069513406156\n - type: dot_accuracy\n value: 99.90069513406156\n - type: dot_ap\n value: 100.0\n - type: dot_f1\n value: 99.95032290114257\n - type: dot_precision\n value: 100.0\n - type: dot_recall\n value: 99.90069513406156\n - type: euclidean_accuracy\n value: 99.90069513406156\n - type: euclidean_ap\n value: 100.0\n - type: euclidean_f1\n value: 99.95032290114257\n - type: euclidean_precision\n value: 100.0\n - type: euclidean_recall\n value: 99.90069513406156\n - type: manhattan_accuracy\n value: 99.90069513406156\n - type: manhattan_ap\n value: 100.0\n - type: manhattan_f1\n value: 99.95032290114257\n - type: manhattan_precision\n value: 100.0\n - type: manhattan_recall\n value: 99.90069513406156\n - type: max_accuracy\n value: 99.90069513406156\n - type: max_ap\n value: 100.0\n - type: max_f1\n value: 99.95032290114257\n task:\n type: PairClassification\n - dataset:\n config: fr\n name: MTEB PawsX (fr)\n revision: 8a04d940a42cd40658986fdd8e3da561533a3646\n split: test\n type: paws-x\n metrics:\n - type: cos_sim_accuracy\n value: 70.8\n - type: cos_sim_ap\n value: 73.7671529695957\n - type: cos_sim_f1\n value: 68.80964339527875\n - type: cos_sim_precision\n value: 62.95955882352941\n - type: cos_sim_recall\n value: 75.85825027685493\n - type: dot_accuracy\n value: 70.8\n - type: dot_ap\n value: 73.78345265366947\n - type: dot_f1\n value: 68.80964339527875\n - type: dot_precision\n value: 62.95955882352941\n - type: dot_recall\n value: 75.85825027685493\n - type: euclidean_accuracy\n value: 70.8\n - type: euclidean_ap\n value: 73.7671529695957\n - type: euclidean_f1\n value: 68.80964339527875\n - type: euclidean_precision\n value: 62.95955882352941\n - type: euclidean_recall\n value: 75.85825027685493\n - type: manhattan_accuracy\n value: 70.75\n - type: manhattan_ap\n value: 73.78996383615953\n - type: manhattan_f1\n value: 68.79432624113475\n - type: manhattan_precision\n value: 63.39869281045751\n - type: manhattan_recall\n value: 75.1937984496124\n - type: max_accuracy\n value: 70.8\n - type: max_ap\n value: 73.78996383615953\n - type: max_f1\n value: 68.80964339527875\n task:\n type: PairClassification\n - dataset:\n config: default\n name: MTEB SICKFr\n revision: e077ab4cf4774a1e36d86d593b150422fafd8e8a\n split: test\n type: Lajavaness/SICK-fr\n metrics:\n - type: cos_sim_pearson\n value: 84.03253762760392\n - type: cos_sim_spearman\n value: 79.68280105762004\n - type: euclidean_pearson\n value: 80.98265050044444\n - type: euclidean_spearman\n value: 79.68233242682867\n - type: manhattan_pearson\n value: 80.9678911810704\n - type: manhattan_spearman\n value: 79.70264097683109\n task:\n type: STS\n - dataset:\n config: fr\n name: MTEB STS22 (fr)\n revision: eea2b4fe26a775864c896887d910b76a8098ad3f\n split: test\n type: mteb/sts22-crosslingual-sts\n metrics:\n - type: cos_sim_pearson\n value: 80.56896987572884\n - type: cos_sim_spearman\n value: 81.84352499523287\n - type: euclidean_pearson\n value: 80.40831759421305\n - type: euclidean_spearman\n value: 81.84352499523287\n - type: manhattan_pearson\n value: 80.74333857561238\n - type: manhattan_spearman\n value: 82.41503246733892\n task:\n type: STS\n - dataset:\n config: fr\n name: MTEB STSBenchmarkMultilingualSTS (fr)\n revision: 93d57ef91790589e3ce9c365164337a8a78b7632\n split: test\n type: stsb_multi_mt\n metrics:\n - type: cos_sim_pearson\n value: 82.71826762276979\n - type: cos_sim_spearman\n value: 82.25433354916042\n - type: euclidean_pearson\n value: 81.87115571724316\n - type: euclidean_spearman\n value: 82.25322342890107\n - type: manhattan_pearson\n value: 82.11174867527224\n - type: manhattan_spearman\n value: 82.55905365203084\n task:\n type: STS\n - dataset:\n config: default\n name: MTEB SummEvalFr\n revision: b385812de6a9577b6f4d0f88c6a6e35395a94054\n split: test\n type: lyon-nlp/summarization-summeval-fr-p2p\n metrics:\n - type: cos_sim_pearson\n value: 30.659441623392887\n - type: cos_sim_spearman\n value: 30.501134097353315\n - type: dot_pearson\n value: 30.659444768851056\n - type: dot_spearman\n value: 30.501134097353315\n task:\n type: Summarization\n - dataset:\n config: default\n name: MTEB SyntecReranking\n revision: b205c5084a0934ce8af14338bf03feb19499c84d\n split: test\n type: lyon-nlp/mteb-fr-reranking-syntec-s2p\n metrics:\n - type: map\n value: 94.03333333333333\n - type: mrr\n value: 94.03333333333333\n task:\n type: Reranking\n - dataset:\n config: default\n name: MTEB SyntecRetrieval\n revision: 77f7e271bf4a92b24fce5119f3486b583ca016ff\n split: test\n type: lyon-nlp/mteb-fr-retrieval-syntec-s2p\n metrics:\n - type: map_at_1\n value: 79.0\n - type: map_at_10\n value: 87.61\n - type: map_at_100\n value: 87.655\n - type: map_at_1000\n value: 87.655\n - type: map_at_3\n value: 87.167\n - type: map_at_5\n value: 87.36699999999999\n - type: mrr_at_1\n value: 79.0\n - type: mrr_at_10\n value: 87.61\n - type: mrr_at_100\n value: 87.655\n - type: mrr_at_1000\n value: 87.655\n - type: mrr_at_3\n value: 87.167\n - type: mrr_at_5\n value: 87.36699999999999\n - type: ndcg_at_1\n value: 79.0\n - type: ndcg_at_10\n value: 90.473\n - type: ndcg_at_100\n value: 90.694\n - type: ndcg_at_1000\n value: 90.694\n - type: ndcg_at_3\n value: 89.464\n - type: ndcg_at_5\n value: 89.851\n - type: precision_at_1\n value: 79.0\n - type: precision_at_10\n value: 9.9\n - type: precision_at_100\n value: 1.0\n - type: precision_at_1000\n value: 0.1\n - type: precision_at_3\n value: 32.0\n - type: precision_at_5\n value: 19.400000000000002\n - type: recall_at_1\n value: 79.0\n - type: recall_at_10\n value: 99.0\n - type: recall_at_100\n value: 100.0\n - type: recall_at_1000\n value: 100.0\n - type: recall_at_3\n value: 96.0\n - type: recall_at_5\n value: 97.0\n task:\n type: Retrieval\n - dataset:\n config: fr\n name: MTEB XPQARetrieval (fr)\n revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f\n split: test\n type: jinaai/xpqa\n metrics:\n - type: map_at_1\n value: 39.395\n - type: map_at_10\n value: 59.123999999999995\n - type: map_at_100\n value: 60.704\n - type: map_at_1000\n value: 60.760000000000005\n - type: map_at_3\n value: 53.187\n - type: map_at_5\n value: 56.863\n - type: mrr_at_1\n value: 62.083\n - type: mrr_at_10\n value: 68.87299999999999\n - type: mrr_at_100\n value: 69.46900000000001\n - type: mrr_at_1000\n value: 69.48299999999999\n - type: mrr_at_3\n value: 66.8\n - type: mrr_at_5\n value: 67.928\n - type: ndcg_at_1\n value: 62.083\n - type: ndcg_at_10\n value: 65.583\n - type: ndcg_at_100\n value: 70.918\n - type: ndcg_at_1000\n value: 71.72800000000001\n - type: ndcg_at_3\n value: 60.428000000000004\n - type: ndcg_at_5\n value: 61.853\n - type: precision_at_1\n value: 62.083\n - type: precision_at_10\n value: 15.033\n - type: precision_at_100\n value: 1.9529999999999998\n - type: precision_at_1000\n value: 0.207\n - type: precision_at_3\n value: 36.315\n - type: precision_at_5\n value: 25.955000000000002\n - type: recall_at_1\n value: 39.395\n - type: recall_at_10\n value: 74.332\n - type: recall_at_100\n value: 94.729\n - type: recall_at_1000\n value: 99.75500000000001\n - type: recall_at_3\n value: 57.679\n - type: recall_at_5\n value: 65.036\n task:\n type: Retrieval\n---\n\n## gte-Qwen2-1.5B-instruct\n\n**gte-Qwen2-1.5B-instruct** is the latest model in the gte (General Text Embedding) model family. The model is built on [Qwen2-1.5B](https://huggingface.co/Qwen/Qwen2-1.5B) LLM model and use the same training data and strategies as the [gte-Qwen2-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) model. \n\nThe model incorporates several key advancements:\n\n- Integration of bidirectional attention mechanisms, enriching its contextual understanding.\n- Instruction tuning, applied solely on the query side for streamlined efficiency\n- Comprehensive training across a vast, multilingual text corpus spanning diverse domains and scenarios. This training leverages both weakly supervised and supervised data, ensuring the model's applicability across numerous languages and a wide array of downstream tasks.\n\n\n## Model Information\n- Model Size: 1.5B \n- Embedding Dimension: 1536\n- Max Input Tokens: 32k\n\n## Requirements\n```\ntransformers>=4.39.2\nflash_attn>=2.5.6\n```\n## Usage \n\n### Sentence Transformers\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", trust_remote_code=True)\n# In case you want to reduce the maximum length:\nmodel.max_seq_length = 8192\n\nqueries = [\n \"how much protein should a female eat\",\n \"summit define\",\n]\ndocuments = [\n \"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.\",\n \"Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.\",\n]\n\nquery_embeddings = model.encode(queries, prompt_name=\"query\")\ndocument_embeddings = model.encode(documents)\n\nscores = (query_embeddings @ document_embeddings.T) * 100\nprint(scores.tolist())\n```\n\nObserve the [config_sentence_transformers.json](config_sentence_transformers.json) to see all pre-built prompt names. Otherwise, you can use `model.encode(queries, prompt=\"Instruct: ...\\nQuery: \"` to use a custom prompt of your choice.\n\n### Transformers\n\n```python\nimport torch\nimport torch.nn.functional as F\n\nfrom torch import Tensor\nfrom transformers import AutoTokenizer, AutoModel\n\n\ndef last_token_pool(last_hidden_states: Tensor,\n attention_mask: Tensor) -> Tensor:\n left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])\n if left_padding:\n return last_hidden_states[:, -1]\n else:\n sequence_lengths = attention_mask.sum(dim=1) - 1\n batch_size = last_hidden_states.shape[0]\n return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]\n\n\ndef get_detailed_instruct(task_description: str, query: str) -> str:\n return f'Instruct: {task_description}\\nQuery: {query}'\n\n\n# Each query must come with a one-sentence instruction that describes the task\ntask = 'Given a web search query, retrieve relevant passages that answer the query'\nqueries = [\n get_detailed_instruct(task, 'how much protein should a female eat'),\n get_detailed_instruct(task, 'summit define')\n]\n# No need to add instruction for retrieval documents\ndocuments = [\n \"As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.\",\n \"Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments.\"\n]\ninput_texts = queries + documents\n\ntokenizer = AutoTokenizer.from_pretrained('Alibaba-NLP/gte-Qwen2-1.5B-instruct', trust_remote_code=True)\nmodel = AutoModel.from_pretrained('Alibaba-NLP/gte-Qwen2-1.5B-instruct', trust_remote_code=True)\n\nmax_length = 8192\n\n# Tokenize the input texts\nbatch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors='pt')\noutputs = model(**batch_dict)\nembeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])\n\n# normalize embeddings\nembeddings = F.normalize(embeddings, p=2, dim=1)\nscores = (embeddings[:2] @ embeddings[2:].T) * 100\nprint(scores.tolist())\n```\n\n## Evaluation\n\n### MTEB & C-MTEB\n\nYou can use the [scripts/eval_mteb.py](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct/blob/main/scripts/eval_mteb.py) to reproduce the following result of **gte-Qwen2-1.5B-instruct** on MTEB(English)/C-MTEB(Chinese):\n\n| Model Name | MTEB(56) | C-MTEB(35) | MTEB-fr(26) | MTEB-pl(26) | \n|:----:|:---------:|:----------:|:----------:|:----------:|\n| [bge-base-en-1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) | 64.23 | - | - | - |\n| [bge-large-en-1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | 63.55 | - | - | - |\n| [gte-large-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5) | 65.39 | - | - | - |\n| [gte-base-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5) | 64.11 | - | - | - |\n| [mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) | 64.68 | - | - | - |\n| [acge_text_embedding](https://huggingface.co/aspire/acge_text_embedding) | - | 69.07 | - | - |\n| [stella-mrl-large-zh-v3.5-1792d](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d) | - | 68.55 | - | - |\n| [gte-large-zh](https://huggingface.co/thenlper/gte-large-zh) | - | 66.72 | - | - |\n| [multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | 59.45 | 56.21 | - | - |\n| [multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 61.50 | 58.81 | - | - |\n| [e5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) | 66.63 | 60.81 | - | - |\n| [gte-Qwen1.5-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct) | 67.34 | 69.52 | - | - |\n| [NV-Embed-v1](https://huggingface.co/nvidia/NV-Embed-v1) | 69.32 | - | - | - |\n| [**gte-Qwen2-7B-instruct**](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) | **70.24** | **72.05** | **68.25** | **67.86** |\n| [**gte-Qwen2-1.5B-instruct**](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct) | **67.16** | **67.65** | **66.60** | **64.04** |\n\n### GTE Models\n\nThe gte series models have consistently released two types of models: encoder-only models (based on the BERT architecture) and decode-only models (based on the LLM architecture). \n\n| Models | Language | Max Sequence Length | Dimension | Model Size (Memory Usage, fp32) |\n|:-------------------------------------------------------------------------------------:|:--------:|:-----: |:---------:|:-------------------------------:|\n| [GTE-large-zh](https://huggingface.co/thenlper/gte-large-zh) | Chinese | 512 | 1024 | 1.25GB |\n| [GTE-base-zh](https://huggingface.co/thenlper/gte-base-zh) | Chinese | 512 | 512 | 0.41GB |\n| [GTE-small-zh](https://huggingface.co/thenlper/gte-small-zh) | Chinese | 512 | 512 | 0.12GB |\n| [GTE-large](https://huggingface.co/thenlper/gte-large) | English | 512 | 1024 | 1.25GB |\n| [GTE-base](https://huggingface.co/thenlper/gte-base) | English | 512 | 512 | 0.21GB |\n| [GTE-small](https://huggingface.co/thenlper/gte-small) | English | 512 | 384 | 0.10GB |\n| [GTE-large-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5) | English | 8192 | 1024 | 1.74GB |\n| [GTE-base-en-v1.5](https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5) | English | 8192 | 768 | 0.51GB |\n| [GTE-Qwen1.5-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct) | Multilingual | 32000 | 4096 | 26.45GB |\n| [GTE-Qwen2-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) | Multilingual | 32000 | 3584 | 26.45GB |\n| [GTE-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct) | Multilingual | 32000 | 1536 | 6.62GB |\n\n## Citation\n\nIf you find our paper or models helpful, please consider cite:\n\n```\n@article{li2023towards,\n title={Towards general text embeddings with multi-stage contrastive learning},\n author={Li, Zehan and Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Pengjun and Zhang, Meishan},\n journal={arXiv preprint arXiv:2308.03281},\n year={2023}\n}\n```\n\n\n",
"related_quantizations": []
},
"tags": [
"gguf",
"arxiv:2308.03281",
"endpoints_compatible",
"region:us",
"conversational"
],
"likes": 0,
"downloads": 112,
"gated": false,
"private": false,
"last_modified": "2024-07-14T10:05:13.000Z",
"created_at": "2024-07-14T08:23:13.000Z",
"pipeline_tag": "",
"library_name": ""
}
Source payload excerpt (from Hugging Face API)
{
"_id": "66938af16ad20abb1dc5aaab",
"id": "RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf",
"modelId": "RichardErkhov/Alibaba-NLP_-_gte-Qwen2-1.5B-instruct-gguf",
"sha": "8833356f68f44725f6ef0534c69e835eed1f675f",
"createdAt": "2024-07-14T08:23:13.000Z",
"lastModified": "2024-07-14T10:05:13.000Z",
"author": "RichardErkhov",
"downloads": 112,
"likes": 0,
"gated": false,
"private": false,
"pipeline_tag": "",
"library_name": "",
"siblings_count": 24
}