maddes8cht/mosaicml-mpt-7b-8k-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "tags": [
      "Composer",
      "MosaicML",
      "llm-foundry",
      "StreamingDatasets"
    ],
    "datasets": [
      "mc4",
      "c4",
      "togethercomputer/RedPajama-Data-1T",
      "bigcode/the-stack",
      "allenai/s2orc"
    ],
    "inference": false,
    "frontmatter": {
      "license": "apache-2.0",
      "tags": [
        "Composer",
        "MosaicML",
        "llm-foundry",
        "StreamingDatasets"
      ],
      "datasets": [
        "mc4",
        "c4",
        "togethercomputer/RedPajama-Data-1T",
        "bigcode/the-stack",
        "allenai/s2orc"
      ],
      "inference": "false"
    },
    "hero_image_url": "https://maddes8cht.github.io/assets/buttons/Huggingface-banner.jpg",
    "summary": "MPT-7b and MPT-30B are part of the family of Mosaic Pretrained Transformer (MPT) models, which use a modified transformer architecture optimized for efficient training and inference. # About GGUF format gguf is the current file format used by the ggml library. A growing list of Software is using it and can therefore use this model. The core project making use of the ggml library is the llama.cpp project by Georgi Gerganov # Quantization variants There is a bunch of quantized files available to cater to your specific needs. Here's how to choose the best option for you: # Legacy quants Q4_0, Q4_1, Q5_0, Q5_1 and Q8 are legacy quantization types. Nevertheless, they are fully supported, as there are several circumstances that cause certain model not to be compatible with the modern K-quants.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\ntags:\n- Composer\n- MosaicML\n- llm-foundry\n- StreamingDatasets\ndatasets:\n- mc4\n- c4\n- togethercomputer/RedPajama-Data-1T\n- bigcode/the-stack\n- allenai/s2orc\ninference: false\n---\n[![banner](https://maddes8cht.github.io/assets/buttons/Huggingface-banner.jpg)]()\n\nI'm constantly enhancing these model descriptions to provide you with the most relevant and comprehensive information\n\n# mpt-7b-8k - GGUF\n- Model creator: [mosaicml](https://huggingface.co/mosaicml)\n- Original model: [mpt-7b-8k](https://huggingface.co/mosaicml/mpt-7b-8k)\n\nMPT-7b and MPT-30B are part of the family of Mosaic Pretrained Transformer (MPT) models, which use a modified transformer architecture optimized for efficient training and inference.\n\n\n\n# About GGUF format\n\n`gguf` is the current file format used by the [`ggml`](https://github.com/ggerganov/ggml) library.\nA growing list of Software is using it and can therefore use this model.\nThe core project making use of the ggml library is the [llama.cpp](https://github.com/ggerganov/llama.cpp) project by Georgi Gerganov\n\n# Quantization variants\n\nThere is a bunch of quantized files available to cater to your specific needs. Here's how to choose the best option for you:\n\n# Legacy quants\n\nQ4_0, Q4_1, Q5_0, Q5_1 and Q8 are `legacy` quantization types.\nNevertheless, they are fully supported, as there are several circumstances that cause certain model not to be compatible with the modern K-quants.\n## Note:\nNow there's a new option to use K-quants even for previously 'incompatible' models, although this involves some fallback solution that makes them not *real* K-quants. More details can be found in affected model descriptions.\n(This mainly refers to Falcon 7b and Starcoder models)\n\n# K-quants\n\nK-quants are designed with the idea that different levels of quantization in specific parts of the model can optimize performance, file size, and memory load.\nSo, if possible, use K-quants.\nWith a Q6_K, you'll likely find it challenging to discern a quality difference from the original model - ask your model two times the same question and you may encounter bigger quality differences.\n\n\n\n\n---\n\n# Original Model Card:\n# MPT-7B-8k\n\nMPT-7B-8k is a decoder-style transformer pretrained starting from MPT-7B, but updating the sequence length to 8k and training for an additional 500B tokens, resulting in a total of 1.5T tokens of text and code.\nThis model was trained by [MosaicML](https://www.mosaicml.com).\n\nMPT-7B-8k is part of the family of Mosaic Pretrained Transformer (MPT) models, which use a modified transformer architecture optimized for efficient training and inference.\n\nThese architectural changes include performance-optimized layer implementations and the elimination of context length limits by replacing\npositional embeddings with Attention with Linear Biases ([ALiBi](https://arxiv.org/abs/2108.12409)).\nThanks to these modifications, MPT models can be trained with high throughput efficiency and stable convergence.\nMPT models can also be served efficiently with both standard HuggingFace pipelines and NVIDIA's [FasterTransformer](https://github.com/NVIDIA/FasterTransformer).\n\nThis model uses the MosaicML LLM codebase, which can be found in the [llm-foundry repository](https://github.com/mosaicml/llm-foundry). It was trained by MosaicML’s NLP team on the [MosaicML platform](https://www.mosaicml.com/training) for LLM pretraining, finetuning, and inference.\n\n### How is this model different?\n\nMPT-7B-8k is\n\n* **Licensed for the possibility of commercial use.**\n* **Trained on a large amount of data** (1.5T tokens like [XGen](https://huggingface.co/Salesforce/xgen-7b-8k-base) vs. 1T for [LLaMA](https://arxiv.org/abs/2302.13971), 1T for [MPT-7B](https://www.mosaicml.com/blog/mpt-7b), 300B for [Pythia](https://github.com/EleutherAI/pythia), 300B for [OpenLLaMA](https://github.com/openlm-research/open_llama), and 800B for [StableLM](https://github.com/Stability-AI/StableLM)).\n* **Prepared to handle long inputs** thanks to [ALiBi](https://arxiv.org/abs/2108.12409). With ALiBi, the model can extrapolate beyond the 8k training sequence length to up to 10k, and with a few million tokens it can be finetuned to extrapolate much further.\n* **Capable of fast training and inference** via [FlashAttention](https://arxiv.org/pdf/2205.14135.pdf) and [FasterTransformer](https://github.com/NVIDIA/FasterTransformer)\n* **Equipped with highly efficient open-source training code** via the [llm-foundry repository](https://github.com/mosaicml/llm-foundry)\n\n### Models finetuned off MPT-7B-8k:\n\nThe following models are finetuned on MPT-7B-8k:\n\n* [MPT-7B-8k-Instruct](https://huggingface.co/mosaicml/mpt-7b-8k-instruct): a model for long-form instruction following (especially summarization and question-answering).\nBuilt by finetuning MPT-7B-8k on several carefully curated datasets.\n  * License: _CC-BY-SA-3.0_\n\n* [MPT-7B-8k-Chat](https://huggingface.co/mosaicml/mpt-7b-8k-chat): a chatbot-like model for dialogue generation.\nBuilt by finetuning MPT-7B-8k on approximately 1.5B tokens of chat data.\n  * License: _CC-By-NC-SA-4.0_\n\n## Model Date\n\nJuly 18, 2023\n\n## Model License\n\nApache-2.0\n\n## Documentation\n\n* [Blog post: MPT-7B-8k](https://www.mosaicml.com/blog/long-context-mpt-7b-8k)\n* [Codebase (mosaicml/llm-foundry repo)](https://github.com/mosaicml/llm-foundry/)\n* Questions: Feel free to contact us via the [MosaicML Community Slack](https://mosaicml.me/slack)!\n\n\n## How to Use\n\nThis model is best used with the MosaicML [llm-foundry repository](https://github.com/mosaicml/llm-foundry) for training and finetuning.\n\n```python\nimport transformers\nmodel = transformers.AutoModelForCausalLM.from_pretrained(\n  'mosaicml/mpt-7b-8k',\n  trust_remote_code=True\n)\n```\nNote: This model requires that `trust_remote_code=True` be passed to the `from_pretrained` method.\nThis is because we use a custom `MPT` model architecture that is not yet part of the Hugging Face `transformers` package.\n`MPT` includes options for many training efficiency features such as [FlashAttention](https://arxiv.org/pdf/2205.14135.pdf), [ALiBi](https://arxiv.org/abs/2108.12409), [QK LayerNorm](https://arxiv.org/abs/2010.04245), and more.\n\nTo use the optimized [triton implementation](https://github.com/openai/triton) of FlashAttention, you can load the model on GPU (`cuda:0`) with `attn_impl='triton'` and with `bfloat16` precision:\n```python\nimport torch\nimport transformers\n\nname = 'mosaicml/mpt-7b-8k'\n\nconfig = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)\nconfig.attn_config['attn_impl'] = 'triton'\nconfig.init_device = 'cuda:0' # For fast initialization directly on GPU!\n\nmodel = transformers.AutoModelForCausalLM.from_pretrained(\n  name,\n  config=config,\n  torch_dtype=torch.bfloat16, # Load model weights in bfloat16\n  trust_remote_code=True\n)\n```\n\nAlthough the model was trained with a sequence length of 2048, ALiBi enables users to increase the maximum sequence length during finetuning and/or inference. For example:\n\n```python\nimport transformers\n\nname = 'mosaicml/mpt-7b-8k'\n\nconfig = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)\nconfig.max_seq_len = 10000 # (input + output) tokens can now be up to 10000\n\nmodel = transformers.AutoModelForCausalLM.from_pretrained(\n  name,\n  config=config,\n  trust_remote_code=True\n)\n```\n\nThis model was trained with the MPT-7B-8k tokenizer which is identical to the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer.\n\n```python\nfrom transformers import AutoTokenizer\ntokenizer = AutoTokenizer.from_pretrained('mosaicml/mpt-7b-8k')\n```\n\nThe model can then be used, for example, within a text-generation pipeline.  \nNote: when running Torch modules in lower precision, it is best practice to use the [torch.autocast context manager](https://pytorch.org/docs/stable/amp.html).\n\n```python\nfrom transformers import pipeline\n\nwith torch.autocast('cuda', dtype=torch.bfloat16):\n    inputs = tokenizer('Here is a recipe for vegan banana bread:\\n', return_tensors=\"pt\").to('cuda')\n    outputs = model.generate(**inputs, max_new_tokens=100)\n    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n\n# or using the HF pipeline\npipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')\nwith torch.autocast('cuda', dtype=torch.bfloat16):\n    print(\n        pipe('Here is a recipe for vegan banana bread:\\n',\n            max_new_tokens=100,\n            do_sample=True,\n            use_cache=True))\n```\n\n## Model Description\n\nThe architecture is a modification of a standard decoder-only transformer.\n\nThe model has been modified from a standard transformer in the following ways:\n* It uses [FlashAttention](https://arxiv.org/pdf/2205.14135.pdf)\n* It uses [ALiBi (Attention with Linear Biases)](https://arxiv.org/abs/2108.12409) and does not use positional embeddings\n* It does not use biases\n\n\n| Hyperparameter | Value |\n|----------------|-------|\n|n_parameters | 6.7B |\n|n_layers | 32 |\n| n_heads | 32 |\n| d_model | 4096 |\n| vocab size | 50432 |\n| sequence length | 2048 |\n\n\n\n## Training Data\n\n### Streaming Datasets\n\nData was formatted using the MosaicML [StreamingDataset](https://github.com/mosaicml/streaming) library to host our data in object storage and efficiently stream it to our compute cluster during training.\nStreamingDataset obviates the need to download the whole dataset before starting training, and allows instant resumption of training from any point in the dataset.\n\n\n### Data Mix\n\nThe model was trained for ___T tokens. First it was trained for 1T tokens (with batch size 1760 and sequence length 2048) on the following data mix:\n\n#### Data Mix for Original 1T Tokens Used to Train MPT-7B\n\n| Data Source | Number of Tokens in Source | Proportion | Effective Number of Tokens | Epochs |\n|-------------|----------------------------|------------|----------------------------|--------|\n| mC4 3.1.0 - English | 417.99 B | 0.33 | 330 B | 0.14 |\n| C4 - English - SemDedup 80% | 100.42 B | 0.299 | 299 B | 2.98 |\n| RedPajama - CommonCrawl | 878.45 B | 0.1 | 100 B | 0.11 |\n| The Stack - Selected Languages | 463.78 B | 0.1 | 100 B | 0.22 |\n| RedPajama - Wikipedia - En | 4.87 B | 0.04 | 40 B | 8.21 |\n| The Stack - Markdown | 107.07 B | 0.035 | 35 B | 0.33 |\n| S2ORC | 48.85 B | 0.033 | 33 B | 0.68 |\n| RedPajama - Books | 26.02 B | 0.03 | 30B | 1.15 |\n| RedPajama - arXiv | 28.10 B | 0.019 | 19 B | 0.68 |\n| RedPajama - StackExchange | 20.54 B | 0.014 | 14 B |0.68 |\n\n#### Data Mix for Additional 500B Tokens Used to Further Train MPT-7B-8k\n\nWe took 80B tokens from document samples that were longer than 4096 tokens, and 120B tokens with varying document sample lengths that matched the \"baseline\" length distribution for a total of 200B tokens in a single dataset. \nWe then trained MPT-7B for 500B tokens with a maximum sequence length of 8192, resulting in MPT-7B-8k. Since we trained for 500B tokens using 200B tokens, nearly every subset was trained on for exactly 2.5 epochs.\n\n| Sequence Length Distribution | Number of Tokens in Source (Billion) | Proportion | Effective Number of Tokens (Billion) | Epochs |\n|---|---|---|---|---|\n| mC4 3.1.0 - English (200+ words) - Baseline | 33.60 | 16.80% | 84.00 | 2.50 |\n| mC4 3.1.0 - English (200+ words) - ≥4096 tokens | 23.04 | 11.52% | 57.60 | 2.50 |\n| c4 - English - SemDedup 80% - Baseline | 30.12 | 15.06% | 75.30 | 2.50 |\n| c4 - English - SemDedup 80% - ≥4096 tokens | 0.92 | 0.46% | 2.30 | 2.50 |\n| RedPajama - CommonCrawl - Baseline | 8.52 | 4.26% | 21.30 | 2.50 |\n| RedPajama - CommonCrawl - ≥4096 tokens | 12.80 | 6.40% | 32.00 | 2.50 |\n| The Stack - Selected Languages - Baseline | 30.00 | 15.00% | 75.00 | 2.50 |\n| The Stack - Selected Languages - ≥4096 tokens | 10.00 | 5.00% | 25.00 | 2.50 |\n| RedPajama - Wikipedia - Baseline | 3.60 | 1.80% | 9.00 | 2.50 |\n| RedPajama - Wikipedia - ≥4096 tokens | 1.04 | 0.52% | 2.60 | 2.50 |\n| The Stack - Markdown - Baseline | 4.50 | 2.25% | 11.25 | 2.50 |\n| The Stack - Markdown - ≥4096 tokens | 8.00 | 4.00% | 20.00 | 2.50 |\n| Semantic Scholar ORC - Baseline | 3.30 | 1.65% | 8.25 | 2.50 |\n| Semantic Scholar ORC - ≥4096 tokens | 8.00 | 4.00% | 20.00 | 2.50 |\n| RedPajama - Books - Baseline | 3.00 | 1.50% | 7.50 | 2.50 |\n| RedPajama - Books - ≥4096 tokens | 8.00 | 4.00% | 20.00 | 2.50 |\n| RedPajama - arXiv - Baseline | 1.92 | 0.96% | 4.80 | 2.50 |\n| RedPajama - arXiv - ≥4096 tokens | 5.40 | 2.70% | 13.50 | 2.50 |\n| RedPajama - StackExchange - Baseline | 1.44 | 0.72% | 3.60 | 2.50 |\n| RedPajama - StackExchange - ≥4096 tokens | 1.52 | 1.40% | 7.00 | 4.60 |\n| N Training Tokens | 200 | 100.00% | | 2.5 epochs * 200B = 500B tokens |\n\n\n\nSamples for each batch were selected from one of the datasets with the probability specified above.\nThe examples were shuffled within each dataset, and each example was constructed from as many sequences from that dataset as were necessary to fill the 2048 sequence length.\n\nThe data was tokenized using the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer. This BPE tokenizer has a number of desirable characteristics,\nmost of which are relevant for tokenizing code:\n(1) It was trained on a diverse mix of data that includes code (The Pile)\n(2) It applies consistent space delimitation, unlike the GPT2 tokenizer which tokenizes inconsistently depending on the presence of prefix spaces\n(3) It contains tokens for repeated space characters, which allows superior compression of text with large amounts of repeated space characters.\n\nThe model vocabulary size of 50432 was set to be a multiple of 128 (as in [MEGATRON-LM](https://arxiv.org/abs/1909.08053)), model flop utilization (MFU) increased by up to four percentage points.\n\n### Training Configuration\n\nThis model was trained on 440 A100-40GBs for about 9.5 days using the [MosaicML Platform](https://www.mosaicml.com/platform).\nThe model was trained with sharded data parallelism using [FSDP](https://pytorch.org/docs/stable/fsdp.html) and used the [LION](https://arxiv.org/abs/2302.06675) optimizer.\n\n## Limitations and Biases\n\n_The following language is modified from [EleutherAI's GPT-NeoX-20B](https://huggingface.co/EleutherAI/gpt-neox-20b)_\n\nMPT-7B-8k is **not** intended for deployment without finetuning.\nIt should not be used for human-facing interactions without further guardrails and user consent.\n\nMPT-7B-8k can produce factually incorrect output, and should not be relied on to produce factually accurate information.\nMPT-7B-8k was trained on various public datasets.\nWhile great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.\n\n\n## MosaicML Platform\n\nIf you're interested in [training](https://www.mosaicml.com/training) and [deploying](https://www.mosaicml.com/inference) your own MPT or LLMs on the MosaicML Platform, [sign up here](https://www.mosaicml.com/get-started?utm_source=huggingface&utm_medium=referral&utm_campaign=mpt-7b-8k).\n\n## Disclaimer\n\nThe license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.\n\n## Citation\n\nPlease cite this model using the following format:\n\n```\n@online{MosaicML2023Introducing,\n    author    = {MosaicML NLP Team},\n    title     = {Introducing MPT-7B: A New Standard for Open-Source,\n    ly Usable LLMs},\n    year      = {2023},\n    url       = {www.mosaicml.com/blog/mpt-7b},\n    note      = {Accessed: 2023-03-28}, % change this date\n    urldate   = {2023-03-28} % change this date\n}\n```\n\n***End of original Model File***\n---\n\n\n## Please consider to support my work\n**Coming Soon:** I'm in the process of launching a sponsorship/crowdfunding campaign for my work. I'm evaluating Kickstarter, Patreon, or the new GitHub Sponsors platform, and I am hoping for some support and contribution to the continued availability of these kind of models. Your support will enable me to provide even more valuable resources and maintain the models you rely on. Your patience and ongoing support are greatly appreciated as I work to make this page an even more valuable resource for the community.\n\n<center>\n\n[![GitHub](https://maddes8cht.github.io/assets/buttons/github-io-button.png)](https://maddes8cht.github.io)\n[![Stack Exchange](https://stackexchange.com/users/flair/26485911.png)](https://stackexchange.com/users/26485911)\n[![GitHub](https://maddes8cht.github.io/assets/buttons/github-button.png)](https://github.com/maddes8cht)\n[![HuggingFace](https://maddes8cht.github.io/assets/buttons/huggingface-button.png)](https://huggingface.co/maddes8cht)\n[![Twitter](https://maddes8cht.github.io/assets/buttons/twitter-button.png)](https://twitter.com/maddes1966)\n\n</center>",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "Composer",
    "MosaicML",
    "llm-foundry",
    "StreamingDatasets",
    "dataset:mc4",
    "dataset:c4",
    "dataset:togethercomputer/RedPajama-Data-1T",
    "dataset:bigcode/the-stack",
    "dataset:allenai/s2orc",
    "arxiv:2108.12409",
    "arxiv:2302.13971",
    "arxiv:2205.14135",
    "arxiv:2010.04245",
    "arxiv:1909.08053",
    "arxiv:2302.06675",
    "license:apache-2.0",
    "region:us"
  ],
  "likes": 0,
  "downloads": 114,
  "gated": false,
  "private": false,
  "last_modified": "2023-11-01T15:36:50.000Z",
  "created_at": "2023-10-30T11:24:19.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "653f926397b866853623fe59",
  "id": "maddes8cht/mosaicml-mpt-7b-8k-gguf",
  "modelId": "maddes8cht/mosaicml-mpt-7b-8k-gguf",
  "sha": "e2de4b2698a631ffe7e8218092f0bf3fea4a5aa7",
  "createdAt": "2023-10-30T11:24:19.000Z",
  "lastModified": "2023-11-01T15:36:50.000Z",
  "author": "maddes8cht",
  "downloads": 114,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 16
}