richarderkhov/knowledgator_-_qwen-encoder-1.5b-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "",
    "summary": "> LLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It consists of 3 simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. The model can be further fine-tuned to achieve state-of-the-art performance.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nQwen-encoder-1.5B - GGUF\n- Model creator: https://huggingface.co/knowledgator/\n- Original model: https://huggingface.co/knowledgator/Qwen-encoder-1.5B/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Qwen-encoder-1.5B.Q2_K.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q2_K.gguf) | Q2_K | 0.63GB |\n| [Qwen-encoder-1.5B.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.IQ3_XS.gguf) | IQ3_XS | 0.68GB |\n| [Qwen-encoder-1.5B.IQ3_S.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.IQ3_S.gguf) | IQ3_S | 0.71GB |\n| [Qwen-encoder-1.5B.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q3_K_S.gguf) | Q3_K_S | 0.71GB |\n| [Qwen-encoder-1.5B.IQ3_M.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.IQ3_M.gguf) | IQ3_M | 0.72GB |\n| [Qwen-encoder-1.5B.Q3_K.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q3_K.gguf) | Q3_K | 0.77GB |\n| [Qwen-encoder-1.5B.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q3_K_M.gguf) | Q3_K_M | 0.77GB |\n| [Qwen-encoder-1.5B.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q3_K_L.gguf) | Q3_K_L | 0.82GB |\n| [Qwen-encoder-1.5B.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.IQ4_XS.gguf) | IQ4_XS | 0.84GB |\n| [Qwen-encoder-1.5B.Q4_0.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q4_0.gguf) | Q4_0 | 0.87GB |\n| [Qwen-encoder-1.5B.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.IQ4_NL.gguf) | IQ4_NL | 0.88GB |\n| [Qwen-encoder-1.5B.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q4_K_S.gguf) | Q4_K_S | 0.88GB |\n| [Qwen-encoder-1.5B.Q4_K.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q4_K.gguf) | Q4_K | 0.92GB |\n| [Qwen-encoder-1.5B.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q4_K_M.gguf) | Q4_K_M | 0.92GB |\n| [Qwen-encoder-1.5B.Q4_1.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q4_1.gguf) | Q4_1 | 0.95GB |\n| [Qwen-encoder-1.5B.Q5_0.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q5_0.gguf) | Q5_0 | 1.02GB |\n| [Qwen-encoder-1.5B.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q5_K_S.gguf) | Q5_K_S | 1.02GB |\n| [Qwen-encoder-1.5B.Q5_K.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q5_K.gguf) | Q5_K | 1.05GB |\n| [Qwen-encoder-1.5B.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q5_K_M.gguf) | Q5_K_M | 1.05GB |\n| [Qwen-encoder-1.5B.Q5_1.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q5_1.gguf) | Q5_1 | 1.1GB |\n| [Qwen-encoder-1.5B.Q6_K.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q6_K.gguf) | Q6_K | 1.19GB |\n| [Qwen-encoder-1.5B.Q8_0.gguf](https://huggingface.co/RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf/blob/main/Qwen-encoder-1.5B.Q8_0.gguf) | Q8_0 | 1.53GB |\n\n\n\n\nOriginal model description:\n---\nlicense: apache-2.0\ndatasets:\n- wikimedia/wikipedia\nlanguage:\n- en\nlibrary_name: transformers\ntags:\n- LLM2Vec\n- encoder\n- LLM\n- classification\n- NER\n- question-answering\n---\n# LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders\n\n> LLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It consists of 3 simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. The model can be further fine-tuned to achieve state-of-the-art performance.\n- **Repository:** https://github.com/McGill-NLP/llm2vec\n- **Paper:** https://arxiv.org/abs/2404.05961\n\n## Overview:\nThis is a bi-directional version of Qwen2-1.5B trained with masked token prediction on the Wikipedia dataset. Modern decoder models offer several advantages over classical encoders like BERT:\n\nThey are pre-trained on more recent textual corpora\nThey are trained on larger and more diverse datasets\nModern decoders have better support for long-context windows\nFlash-attention support is available for these models\n\nConsidering these benefits, we are excited to release a series of decoder models tuned to work in a bi-directional setting. This approach combines the strengths of modern decoder architectures with the versatility of bi-directional context understanding, potentially opening up new possibilities for various natural language processing tasks, such as NER.\n\nIn comparison to original LLM2Vec we trained all weights of LLama model, it potentially improve bi-directional abilities of the model.\n\n## Installation\n```bash\npip install llm2vec\n```\n\n## Usage\n```python\nfrom llm2vec.models import Qwen2BiModel\n\nimport torch\nfrom transformers import AutoTokenizer\n\n# Loading base Mistral model, along with custom code that enables bidirectional connections in decoder-only LLMs. MNTP LoRA weights are merged into the base model.\ntokenizer = AutoTokenizer.from_pretrained(\n    \"knowledgator/Qwen-encoder-1.5B\"\n)\n\nmodel = Qwen2BiModel.from_pretrained(\"knowledgator/Qwen-encoder-1.5B\")\n\ntext = \"The quick brown fox jumps over the lazy dog.\"\n\ninputs = tokenizer(text, return_tensors=\"pt\", padding=True, truncation=True, max_length=512)\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nmodel = model.to(device)\ninputs = {k: v.to(device) for k, v in inputs.items()}\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nlast_hidden_states = outputs.last_hidden_state\n```\n\nHere's an improved and expanded version of the README snippet:\n\n## Adapting for Different Discriminative Tasks\n\nOur bi-directional LLaMA model can be easily adapted for various discriminative tasks such as text classification, question answering, and token classification. \nTo use these specialized versions, we provide a [fork of LLM2Vec](https://github.com/Knowledgator/llm2vec) with additional functionality.\n\n### Installation\n\nTo get started, clone our fork of LLM2Vec and install it:\n\n```bash\ngit clone https://github.com/Knowledgator/llm2vec.git\ncd llm2vec\npip install -e .\n```\n\nUsing `-e` flag installs the package in editable mode, which is useful for development.\n\n### Usage\n\nHere's how to import and use the models for different tasks:\n\n```python\nfrom llm2vec import (\n    AutoLLMEncoderForSequenceClassification,\n    AutoLLMEncoderForQuestionAnswering,\n    AutoLLMEncoderForTokenClassification\n)\n\n# Load models for different tasks\nclassification_model = AutoLLMEncoderForSequenceClassification.from_pretrained('knowledgator/Qwen-encoder-1.5B')\nquestion_answering_model = AutoLLMEncoderForQuestionAnswering.from_pretrained('knowledgator/Qwen-encoder-1.5B')\ntoken_classification_model = AutoLLMEncoderForTokenClassification.from_pretrained('knowledgator/Qwen-encoder-1.5B')\n```\n\n### Example: Text Classification\n\nHere's a basic example of how to use the model for text classification:\n\n```python\nfrom transformers import AutoTokenizer\n\n# Load tokenizer\ntokenizer = AutoTokenizer.from_pretrained('knowledgator/Qwen-encoder-1.5B')\n\n# Prepare input\ntext = \"This movie is great!\"\ninputs = tokenizer(text, return_tensors=\"pt\")\n\n# Get classification logits\noutputs = classification_model(**inputs)\nlogits = outputs.logits\n\n# The logits can be used with a softmax function to get probabilities\n# or you can use torch.argmax(logits, dim=1) to get the predicted class\n```\n\n### Fine-tuning\n\nTo fine-tune these models on your specific task:\n\n1. Prepare your dataset in a format compatible with HuggingFace's `datasets` library.\n2. Use the `Trainer` class from HuggingFace's `transformers` library to fine-tune the model.\n\nHere's a basic example:\n\n```python\nfrom transformers import Trainer, TrainingArguments\nfrom datasets import load_dataset\n\n# Load your dataset\ndataset = load_dataset(\"your_dataset\")\n\n# Define training arguments\ntraining_args = TrainingArguments(\n    output_dir=\"./results\",\n    num_train_epochs=3,\n    per_device_train_batch_size=8,\n    per_device_eval_batch_size=8,\n    warmup_steps=500,\n    weight_decay=0.01,\n    logging_dir=\"./logs\",\n)\n\n# Initialize Trainer\ntrainer = Trainer(\n    model=classification_model,\n    args=training_args,\n    train_dataset=dataset[\"train\"],\n    eval_dataset=dataset[\"test\"],\n)\n\n# Fine-tune the model\ntrainer.train()\n```\n\n### Contributing\n\nWe welcome contributions! If you have suggestions for improvements or encounter any issues, please open an issue or submit a pull request on our [GitHub repository](https://github.com/Knowledgator/llm2vec).\n\n\n\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2404.05961",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 260,
  "gated": false,
  "private": false,
  "last_modified": "2024-10-03T21:18:02.000Z",
  "created_at": "2024-10-03T18:12:22.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "66fede86d5ede3018af8ddba",
  "id": "RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf",
  "modelId": "RichardErkhov/knowledgator_-_Qwen-encoder-1.5B-gguf",
  "sha": "34c24690389cac423ae4b30cdf68a1480b4c1a03",
  "createdAt": "2024-10-03T18:12:22.000Z",
  "lastModified": "2024-10-03T21:18:02.000Z",
  "author": "RichardErkhov",
  "downloads": 260,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}