jackrong/gpt-5-distill-llama3.2-3b-instruct-gguf overview

{
  "metadata": {},
  "card_data": {
    "tags": [
      "gguf",
      "llama",
      "llama-3.2",
      "text-generation",
      "conversational"
    ],
    "license": "llama3.2",
    "datasets": [
      "Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507",
      "ytz20/LMSYS-Chat-GPT-5-Chat-Response"
    ],
    "language": [
      "en",
      "zh"
    ],
    "base_model": [
      "unsloth/Llama-3.2-3B-Instruct"
    ],
    "frontmatter": {
      "tags": [
        "gguf",
        "llama",
        "llama-3.2",
        "text-generation",
        "conversational"
      ],
      "license": "llama3.2",
      "datasets": [
        "Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507",
        "ytz20/LMSYS-Chat-GPT-5-Chat-Response"
      ],
      "language": [
        "en",
        "zh"
      ],
      "base_model": [
        "unsloth/Llama-3.2-3B-Instruct"
      ]
    },
    "hero_image_url": "https://img.shields.io/badge/Base_Model-Llama--3.2--3B-blue?style=flat&logo=meta&logoColor=white",
    "summary": "!Llama-3.2 !Instruct !GPT-5 **Model Type**: Instruction-tuned Edge LLM (Llama 3.2 Architecture) This model represents a high-efficiency distillation attempt, combining the lightweight, edge-ready architecture of **Llama-3.2-3B** with the high-quality conversational patterns of **GPT-5**. By filtering for \"normal\" (flawless) responses from the LMSYS dataset, this model aims to deliver flagship-level instruction following in a 3B parameter package. ---",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\ntags:\n- gguf\n- llama\n- llama-3.2\n- text-generation\n- conversational\nlicense: llama3.2\ndatasets:\n- Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507\n- ytz20/LMSYS-Chat-GPT-5-Chat-Response\nlanguage:\n- en\n- zh\nbase_model:\n- unsloth/Llama-3.2-3B-Instruct\n---\n\n# GPT-5-Distill-llama3.2-3B-Instruct\n\n![Llama-3.2](https://img.shields.io/badge/Base_Model-Llama--3.2--3B-blue?style=flat&logo=meta&logoColor=white)\n![Instruct](https://img.shields.io/badge/Type-Instruction--Tuned-green?style=flat)\n![GPT-5](https://img.shields.io/badge/Distillation-GPT--5_Responses-8A2BE2?style=flat)\n\n**Model Type**: Instruction-tuned Edge LLM (Llama 3.2 Architecture)\n- **Base Model**: `unsloth/Llama-3.2-3B-Instruct`\n- **Parameters**: ~3.2B (Optimized for Edge/Consumer GPU)\n- **Training Method**:\n  - **SFT (Supervised Fine-Tuning)** using Unsloth & TRL\n  - **Knowledge Distillation**: Trained on GPT-5 responses to mimic superior reasoning and tone\n  - **LoRA Config**: r=32, alpha=32, targeting all linear projections\n- **Max Context Length**: **32K tokens** (`max_seq_length = 32768`)\n- **Quantization**: Native GGUF support (Q4_K_M, Q8_0, FP16) provided\n\nThis model represents a high-efficiency distillation attempt, combining the lightweight, edge-ready architecture of **Llama-3.2-3B** with the high-quality conversational patterns of **GPT-5**. By filtering for \"normal\" (flawless) responses from the LMSYS dataset, this model aims to deliver flagship-level instruction following in a 3B parameter package.\n\n---\n\n## 2. Intended Use Cases\n\n### ✅ Recommended:\n- **On-Device Chat**: Perfect for laptops, phones, and low-VRAM GPUs due to small size.\n- **Reasoning & Explanations**: Distilled GPT-5 logic helps in providing clearer answers.\n- **Summarization & Rewriting**: Inherits strong English/Chinese capabilities from the dataset mix.\n- **RAG Applications**: 32K context window allows for processing moderate-sized documents.\n\n### ⚠️ Not Suitable For:\n- **Math/Complex Coding**: While capable, 3B models have limitations compared to 70B+ models in complex logic.\n- **High-Stakes Medical/Legal Advice**: Outputs should always be verified.\n- **Hallucination-Free Tasks**: Small models may still hallucinate facts.\n\n---\n\n## 3. Training Data & Methodology\n\nThe model was trained on a curated mix of **~104,000 high-quality samples**:\n\n### (1) ds1: ShareGPT-Qwen3 Instruction Mix (~3,900 samples)\n- **Source**: `Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507`\n- **Role**: Provides diverse, multi-turn instruction following capabilities, enhancing the model's ability to handle complex prompts (English & Chinese mixed).\n\n### (2) ds2: LMSYS GPT-5 Teacher Responses (~100,000 samples)\n- **Source**: `ytz20/LMSYS-Chat-GPT-5-Chat-Response`\n- **Filtering Logic**:\n  - Applied rigorous filtering: `flaw == \"normal\"` (Removed hallucinations, refusals, and bad formatting).\n  - Only clean, high-quality \"Teacher\" responses were used for distillation.\n- **Role**: Imparts the \"GPT-5\" conversational style, politeness, and reasoning structure to the smaller Llama model.\n\n### Training Configuration:\n- **Framework**: Unsloth + Hugging Face TRL\n- **Loss Masking**: `train_on_responses_only` was enabled (Model learns to generate answers, not questions).\n- **Optimizer**: AdamW 8-bit for efficiency.\n- **Precision**: Trained in 4-bit, exported to 16-bit and GGUF.\n\n---\n\n## 4. Prompt Format (Llama 3.2 Standard)\n\nThis model uses the standard **Llama 3 / 3.2** prompt template.\n\n```text\n<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{Your Prompt Here}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n````\n\n**Python Inference Example:**\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nimport torch\n\nmodel_id = \"Jackrong/GPT-5-Distill-llama3.2-3B-Instruct\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    torch_dtype=torch.bfloat16,\n    device_map=\"auto\",\n)\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"Explain quantum mechanics to a 5-year-old.\"},\n]\n\ninput_ids = tokenizer.apply_chat_template(\n    messages,\n    add_generation_prompt=True,\n    return_tensors=\"pt\"\n).to(model.device)\n\noutputs = model.generate(\n    input_ids,\n    max_new_tokens=512,\n    temperature=0.7,\n    do_sample=True\n)\n\nprint(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))\n```\n\n-----\n\n## 5\\. Key Features Summary\n\n| Feature | Description |\n|--------|-------------|\n| **Super Lightweight** | 3B Parameters. Runs on almost any modern consumer hardware. |\n| **GPT-5 Distilled** | Learned from 100k+ clean GPT-5 outputs for superior tone. |\n| **Long Context** | Supports up to **32k context**, great for long conversations. |\n| **GGUF Ready** | Available in `q4_k_m` (very fast) and `q8_0` quantizations. |\n\n-----\n\n## 6\\. Acknowledgements\n\n  - **Unsloth**: For the 2x faster training and 4-bit loading capabilities.\n  - **LMSYS Org**: For providing the GPT-5 response dataset.\n  - **Meta AI**: For the robust Llama-3.2 base model.\n\nThis project is an open research effort to bring \"Big Model Intelligence\" to \"Small Model Footprints.\"\n\n```\n```",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "llama",
    "llama-3.2",
    "text-generation",
    "conversational",
    "en",
    "zh",
    "dataset:Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507",
    "dataset:ytz20/LMSYS-Chat-GPT-5-Chat-Response",
    "base_model:unsloth/Llama-3.2-3B-Instruct",
    "base_model:quantized:unsloth/Llama-3.2-3B-Instruct",
    "license:llama3.2",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 0,
  "downloads": 337,
  "gated": false,
  "private": false,
  "last_modified": "2025-11-30T15:11:51.000Z",
  "created_at": "2025-11-30T03:58:59.000Z",
  "pipeline_tag": "text-generation",
  "library_name": ""
}

{
  "_id": "692bc103624bf399dc64b948",
  "id": "Jackrong/GPT-5-Distill-llama3.2-3B-Instruct-GGUF",
  "modelId": "Jackrong/GPT-5-Distill-llama3.2-3B-Instruct-GGUF",
  "sha": "4c8817325149dec77119099557c7ccb9bafce5d8",
  "createdAt": "2025-11-30T03:58:59.000Z",
  "lastModified": "2025-11-30T15:11:51.000Z",
  "author": "Jackrong",
  "downloads": 337,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "",
  "siblings_count": 5
}