richarderkhov/euclaise_-_memphis-cot-3b-gguf overview

{
  "metadata": {},
  "card_data": {
    "frontmatter": {},
    "hero_image_url": "https://cdn-uploads.huggingface.co/production/uploads/64137e2150358a805203cbac/DlTWku8gant1yx6NaxqJX.png",
    "summary": "Quantization made by Richard Erkhov. Github Discord Request more models Memphis-CoT-3B - GGUF | Name | Quant method | Size | | ---- | ---- | ---- | | Memphis-CoT-3B.Q2_K.gguf | Q2_K | 1.01GB | | Memphis-CoT-3B.IQ3_XS.gguf | IQ3_XS | 1.11GB | | Memphis-CoT-3B.IQ3_S.gguf | IQ3_S | 1.17GB | | Memphis-CoT-3B.Q3_K_S.gguf | Q3_K_S | 1.17GB | | Memphis-CoT-3B.IQ3_M.gguf | IQ3_M | 1.23GB | | Memphis-CoT-3B.Q3_K.gguf | Q3_K | 1.3GB | | Memphis-CoT-3B.Q3_K_M.gguf | Q3_K_M | 1.3GB | | Memphis-CoT-3B.Q3_K_L.gguf | Q3_K_L | 1.4GB | | Memphis-CoT-3B.IQ4_XS.gguf | IQ4_XS | 1.43GB | | Memphis-CoT-3B.Q4_0.gguf | Q4_0 | 1.5GB | | Memphis-CoT-3B.IQ4_NL.gguf | IQ4_NL | 1.51GB | | Memphis-CoT-3B.Q4_K_S.gguf | Q4_K_S | 1.51GB | | Memphis-CoT-3B.Q4_K.gguf | Q4_K | 1.59GB | | Memphis-CoT-3B.Q4_K_M.gguf | Q4_K_M | 1.59GB | | Memphis-CoT-3B.Q4_1.gguf | Q4_1 | 1.65GB | | Memphis-CoT-3B.Q5_0.gguf | Q5_0 | 1.81GB | | Memphis-CoT-3B.Q5_K_S.gguf | Q5_K_S | 1.81GB | | Memphis-CoT-3B.Q5_K.gguf | Q5_K | 1.86GB | | Memphis-CoT-3B.Q5_K_M.gguf | Q5_K_M | 1.86GB | | Memphis-CoT-3B.Q5_1.gguf | Q5_1 | 1.96GB | | Memphis-CoT-3B.Q6_K.gguf | Q6_K | 2.14GB | | Memphis-CoT-3B.Q8_0.gguf | Q8_0 | 2.77GB | Original model description: --- license: cc-by-sa-3.0 library_name: transformers tags: datasets: metrics: base_model: stabilityai/stablelm-3b-4e1t --- *Now with a training bug fixed!* !image/png Memphis-CoT is a finetune of StableLM 3b 4e1t on TinyCoT, SciCoT, along with reddit-instruct (subset to 5000 examples, excluding posts with brackets in the title) and a curated subset of oasst2. **Memphis was trained *only* on human data! No GPT generations here.** Finetuning was performed using my supertrainer2000 framework, using my Adalite optimizer.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "Quantization made by Richard Erkhov.\n\n[Github](https://github.com/RichardErkhov)\n\n[Discord](https://discord.gg/pvy7H8DZMG)\n\n[Request more models](https://github.com/RichardErkhov/quant_request)\n\n\nMemphis-CoT-3B - GGUF\n- Model creator: https://huggingface.co/euclaise/\n- Original model: https://huggingface.co/euclaise/Memphis-CoT-3B/\n\n\n| Name | Quant method | Size |\n| ---- | ---- | ---- |\n| [Memphis-CoT-3B.Q2_K.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q2_K.gguf) | Q2_K | 1.01GB |\n| [Memphis-CoT-3B.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.IQ3_XS.gguf) | IQ3_XS | 1.11GB |\n| [Memphis-CoT-3B.IQ3_S.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.IQ3_S.gguf) | IQ3_S | 1.17GB |\n| [Memphis-CoT-3B.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q3_K_S.gguf) | Q3_K_S | 1.17GB |\n| [Memphis-CoT-3B.IQ3_M.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.IQ3_M.gguf) | IQ3_M | 1.23GB |\n| [Memphis-CoT-3B.Q3_K.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q3_K.gguf) | Q3_K | 1.3GB |\n| [Memphis-CoT-3B.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q3_K_M.gguf) | Q3_K_M | 1.3GB |\n| [Memphis-CoT-3B.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q3_K_L.gguf) | Q3_K_L | 1.4GB |\n| [Memphis-CoT-3B.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.IQ4_XS.gguf) | IQ4_XS | 1.43GB |\n| [Memphis-CoT-3B.Q4_0.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q4_0.gguf) | Q4_0 | 1.5GB |\n| [Memphis-CoT-3B.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.IQ4_NL.gguf) | IQ4_NL | 1.51GB |\n| [Memphis-CoT-3B.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q4_K_S.gguf) | Q4_K_S | 1.51GB |\n| [Memphis-CoT-3B.Q4_K.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q4_K.gguf) | Q4_K | 1.59GB |\n| [Memphis-CoT-3B.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q4_K_M.gguf) | Q4_K_M | 1.59GB |\n| [Memphis-CoT-3B.Q4_1.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q4_1.gguf) | Q4_1 | 1.65GB |\n| [Memphis-CoT-3B.Q5_0.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q5_0.gguf) | Q5_0 | 1.81GB |\n| [Memphis-CoT-3B.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q5_K_S.gguf) | Q5_K_S | 1.81GB |\n| [Memphis-CoT-3B.Q5_K.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q5_K.gguf) | Q5_K | 1.86GB |\n| [Memphis-CoT-3B.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q5_K_M.gguf) | Q5_K_M | 1.86GB |\n| [Memphis-CoT-3B.Q5_1.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q5_1.gguf) | Q5_1 | 1.96GB |\n| [Memphis-CoT-3B.Q6_K.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q6_K.gguf) | Q6_K | 2.14GB |\n| [Memphis-CoT-3B.Q8_0.gguf](https://huggingface.co/RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf/blob/main/Memphis-CoT-3B.Q8_0.gguf) | Q8_0 | 2.77GB |\n\n\n\n\nOriginal model description:\n---\nlicense: cc-by-sa-3.0\nlibrary_name: transformers\ntags:\n- supertrainer2000\n- human-data\ndatasets:\n- euclaise/TinyCoT\n- euclaise/reddit-instruct\n- sablo/oasst2_curated\n- euclaise/SciCoT\nmetrics:\n- accuracy\nbase_model: stabilityai/stablelm-3b-4e1t\n---\n\n\n\n*Now with a training bug fixed!*\n\n\n\n![image/png](https://cdn-uploads.huggingface.co/production/uploads/64137e2150358a805203cbac/DlTWku8gant1yx6NaxqJX.png)\n\nMemphis-CoT is a finetune of [StableLM 3b 4e1t](stabilityai/stablelm-3b-4e1t) on [TinyCoT](https://huggingface.co/datasets/euclaise/TinyCoT), [SciCoT](https://huggingface.co/datasets/euclaise/SciCoT), along with [reddit-instruct](https://huggingface.co/datasets/euclaise/reddit-instruct) (subset to 5000 examples, excluding posts with brackets in the title) and a [curated](https://huggingface.co/datasets/sablo/oasst2_curated) subset of [oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2).\n\n**Memphis was trained *only* on human data! No GPT generations here.**\n\nFinetuning was performed using my [supertrainer2000](https://github.com/euclaise/supertrainer2000) framework, using my Adalite optimizer.\n\n\n## Training Procedure\nI finetuned the model using an iterative rationale-bootstrapping procedure inspired by [STaR](https://research.google/pubs/star-self-taught-reasoner-bootstrapping-reasoning-with-reasoning/) and [SPIN](https://arxiv.org/abs/2401.01335)\n\nFirst, I finetuned the model on all the datasets using a [MixCE](https://arxiv.org/abs/2305.16958) loss and [NEFTune](https://arxiv.org/abs/2310.05914), for 2 epochs.\n\nI then performed the following steps 3 times:\n1. Generate responses for each question in TinyCoT using the current model, check each response for correctness, and create a dataset of (correct, incorrect) pairs. Extra values are discarded, such that each correct and incorrect response is unique.\n2. Finetune the model for 1 epoch using a ranking loss over length-normalized log-probabilities of each sequence, similar to [Preference Ranking Optimization](https://arxiv.org/abs/2306.17492), comparing the correct vs incorrect generated response. Additionally, a standard CE loss over the chosen completion was included.\n\nThis should be more efficient than either STaR or SPIN, as it uses a ranking loss rather than rejection sampling (unlike STaR), and verifies correctness instead of assuming all model responses are incorrect (unlike SPIN).\n\nTo prevent excessive drift, I kept the model weights as a moving average: After each generate+train cycle, I interpolated between the previous model weights and the updated weights using spherical linear interpolation (SLERP), with an interpolation factor of 0.99.\n\n## Prompt formats\n\nThe format for reddit-instruct and oasst2 was:\n\n```\n### User:\n[insert instruction here]\n### Assistant:\n[insert response here]\n### User:\n...\n```\n\nThe format for TinyCoT was:\n```\n### User:\n[insert instruction here]\n### Rationale:\n[insert reasoning here]\n### Answer:\n[insert direct answer here]\n```\n\n## Benchmarks\n\n| Model                                                                  | Size   | Data                | Method        | GSM8K (5-shot) | AGIEval (English/Nous subset, acc_norm) | BIG Bench Hard (CoT, few-shot*) |\n|:-----------------------------------------------------------------------|--------|:--------------------|---------------|:---------------|:----------------------------------------|:------------------------------  |\n| [StableLM 3B Base](https://hf.co/stabilityai/stablelm-3b-4e1t)       | 3B     | Base                  | Base          |    2.05%       | 25.14%                                  |  36.75%                         |\n| [StableHermes 3B](https://hf.co/cxllin/StableHermes-3b)                | 3B     | GPT                 | SFT           |    3.64%       | 24.31%                                  | **37.28%**                      |\n| [MPT 7B Instruct](https://hf.co/mosaicml/mpt-7b-instruct)              | **7B** | **Human**+Anthropic | SFT           |    2.05%       | 24.12%                                  | 11.01%                          |\n| [OpenLLaMA 7B v2 open-instruct](http://hf.co/VMware/open-llama-7b-v2-open-instruct) | **7B** | **Human** (nearly: ecqa is an exception) | SFT | 8.64% | 23.21%                   | 29.84%                          |\n| [StableLM Zephyr 3B](https://hf.co/stabilityai/stablelm-zephyr-3b)     | 3B     | GPT                 | DPO           |    possibly contaminated (45.72%)  | **33.31%**          | 0.91%                           |\n| [LIMA LLaMA 2 7B](https://huggingface.co/heegyu/LIMA2-7b-hf)           | **7B** | **Human**           | SFT           | 4.55%          |  24.55%                                 | 36.29%                          |\n| [**Memphis-CoT 3B**](https://hf.co/euclaise/Memphis-CoT-3B)            | 3B     | **Human**           | Self-teaching |    **18.8%**       | *27.22%*                            | *36.92%*                      |\n\n*5-shot, as performed automatically by LM Evaluation Harness bbh_cot_fewshot even with num_fewshot=0\n\nMemphis outperforms other primarily-human-data models that are over twice its size, along with SFT models of its size, and trades with the Zephyr DPO model. That said, Zephyr uses synthetic data, and *much* more of it.\n\nNote that BBH results have wide SEs, sometimes even exceeding 16%.\n\n\nIt is unclear why Zephyr performs so poorly on BBH. Perhaps it is overfit, or maybe there was an issue with vllm.\n\nNotes:\n- Evaluations were performed using the `agieval` branch of [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) (commit `0bef5c9c273b1c2f68e6018d4bb9c32b9aaff298`), using the `vllm` model.\n- I tried to find human-data-trained StableLM models, but couldn't find any. I did find a few OpenLLaMA models, but they wouldn't load with LM Eval Harness and vllm. (I believe this can be fixed by changing the xformers backend, but I'm too lazy for that)\n- OpenLLaMA 7B v2 open-instruct is a particularly relevant comparison, as it was trained on a *very* similar dataset.\n\n## Hyperparameters\n\nFor the initial supervised finetuning step:\n- Adalite optimizer, default hyperparameters of supertrainer2000 unless otherwise specified\n- Lambda (Adalite's analogue to weight decay, see [here](https://arxiv.org/abs/2103.06583) for details) of 0.01\n- LR of 1e-5\n- MixCE ratio of 0.75\n- Sequence length of 4096\n- Cosine decay with a 20% warmup\n- Frozen embeddings\n- No training on inputs\n- Accumulated batch size of 128\n- NEFTune with an alpha of 10\n\nFor the generations:\n- Generated using the current git version of `vllm`\n- N=8\n- Temperature of 0.5\n- `top_p` of 0.8\n- Maximum of 512 generated tokens, discarding responses that do not have a valid rationale and answer\n\nFor the rank finetuning:\n- Adalite optimizer, default hyperparameters of supertrainer2000 unless otherwise specified\n- Lambda of 0.01\n- LR of 5e-7\n- Rank loss weight of 0.25\n- Sequence length of 1024\n- Cosine schedule with 10% warmup\n- Frozen embeddings\n- No training on inputs\n- Accumulated batch size of 128\n- NEFTune with an alpha of 10\n\n\nAdditional thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more quants, at much higher speed, than I would otherwise be able to.",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "arxiv:2401.01335",
    "arxiv:2305.16958",
    "arxiv:2310.05914",
    "arxiv:2306.17492",
    "arxiv:2103.06583",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 123,
  "gated": false,
  "private": false,
  "last_modified": "2024-10-21T10:03:18.000Z",
  "created_at": "2024-10-21T09:33:07.000Z",
  "pipeline_tag": "",
  "library_name": ""
}

{
  "_id": "67161fd305c3d282deabe856",
  "id": "RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf",
  "modelId": "RichardErkhov/euclaise_-_Memphis-CoT-3B-gguf",
  "sha": "c105d48a38f55536a1e62cb78668efd17d74e927",
  "createdAt": "2024-10-21T09:33:07.000Z",
  "lastModified": "2024-10-21T10:03:18.000Z",
  "author": "RichardErkhov",
  "downloads": 123,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "",
  "siblings_count": 24
}