aaryank/kimi-linear-48b-a3b-instruct-gguf overview

{
  "metadata": {},
  "card_data": {
    "base_model": "moonshotai/Kimi-Linear-48B-A3B-Instruct",
    "base_model_relation": "quantized",
    "language": [
      "en",
      "zh"
    ],
    "library_name": "gguf",
    "license": "mit",
    "pipeline_tag": "text-generation",
    "tags": [
      "text-generation-inference",
      "linear-attention",
      "rnn",
      "chat",
      "gguf",
      "experimental"
    ],
    "frontmatter": {
      "base_model": "moonshotai/Kimi-Linear-48B-A3B-Instruct",
      "base_model_relation": "quantized",
      "language": [
        "en",
        "zh"
      ],
      "library_name": "gguf",
      "license": "mit",
      "pipeline_tag": "text-generation",
      "tags": [
        "text-generation-inference",
        "linear-attention",
        "rnn",
        "chat",
        "gguf",
        "experimental"
      ]
    },
    "hero_image_url": "https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/resolve/main/figures/logo.png",
    "summary": "Kimi Linear: An Expressive, Efficient Attention Architecture    Paper   Code   Original Model    > [!IMPORTANT] > **I am currently looking for open positions!** 🤗 > If you find this model useful or are looking for a talented AI/LLM Engineer, please reach out to me on LinkedIn: **Aaryan Kapoor**. > [!CAUTION] > **Experimental Build Required** 🚧 > This model utilizes the **Kimi Delta Attention (KDA)** architecture, which is not yet supported in the main branch of llama.cpp. > > To run this GGUF, you **must** compile llama.cpp from **PR #17592**. > Attempting to run this on a standard build will result in errors.   Some test prompts :)",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nbase_model: moonshotai/Kimi-Linear-48B-A3B-Instruct\nbase_model_relation: quantized\nlanguage:\n  - en\n  - zh\nlibrary_name: gguf\nlicense: mit\npipeline_tag: text-generation\ntags:\n  - text-generation-inference\n  - linear-attention\n  - rnn\n  - chat\n  - gguf\n  - experimental\n---\n\n# Kimi-Linear-48B-Instruct-GGUF\n\n<div align=\"center\">\n  <img src=\"https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/resolve/main/figures/logo.png\" width=\"5%\" style=\"vertical-align: middle;\"/>\n  <b>Kimi Linear: An Expressive, Efficient Attention Architecture</b>\n</div>\n\n<div align=\"center\">\n  <a href=\"https://huggingface.co/papers/2510.26692\" style=\"margin: 0 8px;\">\n    <b>Paper</b>\n  </a>\n  <a href=\"https://github.com/MoonshotAI/Kimi-Linear\" style=\"margin: 0 8px;\">\n    <b>Code</b>\n  </a>\n  <a href=\"https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct\" style=\"margin: 0 8px;\">\n    <b>Original Model</b>\n  </a>\n</div>\n\n<br>\n\n> [!IMPORTANT]\n> **I am currently looking for open positions!** 🤗\n> If you find this model useful or are looking for a talented AI/LLM Engineer, please reach out to me on LinkedIn: **[Aaryan Kapoor](https://www.linkedin.com/in/theaaryankapoor/)**.\n\n> [!CAUTION]\n> **Experimental Build Required** 🚧\n> This model utilizes the **Kimi Delta Attention (KDA)** architecture, which is not yet supported in the main branch of `llama.cpp`.\n>\n> To run this GGUF, you **must** compile `llama.cpp` from **[PR #17592](https://github.com/ggml-org/llama.cpp/pull/17592)**.\n> Attempting to run this on a standard build will result in errors.\n\n<div align=\"center\">\n  <img src=\"https://cdn-uploads.huggingface.co/production/uploads/64e1a459ff3fd4fd8eedb456/IlxKYaUsl15644y-k8GNo.png\" width=\"70%\" style=\"vertical-align: middle;\"/>\n  <b>Some test prompts :)</b>\n</div>\n\n## Description\n\nThis repository contains **experimental GGUF** format model files for [Moonshot AI's Kimi Linear 48B](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct).\n\n**Kimi Linear** is a hybrid linear attention architecture designed to outperform traditional full attention methods in long-context and scaling regimes. It uses **Kimi Delta Attention (KDA)** and a hybrid architecture (3:1 KDA-to-MLA ratio) to reduce memory usage and boost throughput by up to 6x on long sequences.\n\n**Performance & Architecture.**\nThis model is currently quantized to **Q2_K** (and others) to fit on consumer hardware while testing the architecture's correctness. Despite the aggressive quantization, initial tests show the logic and reasoning capabilities remain intact.\n\n| Feature | Kimi Linear Specification |\n|:---|:---|\n| **Architecture** | Hybrid Linear Attention (MoE + MLA + KDA) |\n| **Context Length** | 1M Tokens (Supported by architecture) |\n| **Params** | 48B Total / 3B Activated |\n| **Throughput** | ~6.3x faster TPOT compared to MLA at 1M context |\n| **MMLU-Pro** | 51.0 (4k context) |\n| **RULER** | 84.3 (128k context, Pareto-optimal) |\n\n## How to Run (llama.cpp)\n\n**Prerequisite:** You must clone and build the specific PR branch:\n\n```bash\ngit clone https://github.com/ggml-org/llama.cpp\ncd llama.cpp\ngit fetch origin pull/17592/head:pr-17592\ngit checkout pr-17592\nmake -j\n```\n\n### 1. CLI Inference (Interactive Chat)\n\n```bash\n./llama-cli -m Kimi-Linear-48B-Instruct.Q2_K.gguf \\\n  -n 2048 \\                  # Adjust generation limit\n  -c 8192 \\                  # Context window (Model supports up to 1M)\n  --temp 0.8 \\               # Recommended temperature\n  --top-p 0.9 \\\n  -ngl 99 \\                  # Offload all layers to GPU\n  -p \"<|im_start|>user\\nHello, who are you?<|im_end|>\\n<|im_start|>assistant\\n\" \\\n  -cnv\n```\n\n> **Note:** The current GGUF implementation successfully mitigates previous \"state collapse\" issues found in early development.\n\n### 2. Server Mode (API)\n\nRunning a persistent server is recommended for this size model to avoid reloading times.\n\n```bash\n./llama-server -m Kimi-Linear-48B-Instruct.Q2_K.gguf \\\n  --port 8080 \\\n  -ngl 99 \\\n  -c 8192 \\\n  --alias kimi\n```\n\n### Hardware Requirements\n\n*   **Full GPU Offloading (`-ngl 99`):**\n    *   **Q4_K_M:** Requires ~28GB VRAM (e.g., A100, A6000, or Mac Studio M2/M3 Max).\n    *   **Q2_K:** Requires ~16-18GB VRAM (Fits on **RTX 3090 / 4090**).\n*   **Split Offloading:**\n    *   If you have less VRAM (e.g., 12GB), use `-ngl` with a lower number (e.g., `-ngl 20`) to split layers between GPU and CPU RAM.\n\n**Default Settings**\n\n* temperature: `0.8`\n* top-p: `0.9`\n* repeat-penalty: `1.05` (Optional, if repetition occurs)\n\n### CLI Example\n\n```bash\n./llama-cli -m Kimi-Linear-48B-Instruct.Q2_K.gguf \\\n  -c 8192 \\\n  --temp 0.8 \\\n  --top-p 0.9 \\\n  -p \"<|im_start|>user\\nWrite a Python script to calculate Fibonacci numbers.<|im_end|>\\n<|im_start|>assistant\\n\" \\\n  -cnv\n```",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "text-generation-inference",
    "linear-attention",
    "rnn",
    "chat",
    "experimental",
    "text-generation",
    "conversational",
    "en",
    "zh",
    "arxiv:2510.26692",
    "base_model:moonshotai/Kimi-Linear-48B-A3B-Instruct",
    "base_model:quantized:moonshotai/Kimi-Linear-48B-A3B-Instruct",
    "license:mit",
    "endpoints_compatible",
    "region:us"
  ],
  "likes": 10,
  "downloads": 128,
  "gated": false,
  "private": false,
  "last_modified": "2025-12-31T19:20:22.000Z",
  "created_at": "2025-12-25T22:57:00.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "gguf"
}

{
  "_id": "694dc13c329f48253227ec0d",
  "id": "AaryanK/Kimi-Linear-48B-A3B-Instruct-GGUF",
  "modelId": "AaryanK/Kimi-Linear-48B-A3B-Instruct-GGUF",
  "sha": "0ef14054c9ceaaba51b6afb048a7a7d931a6b1cf",
  "createdAt": "2025-12-25T22:57:00.000Z",
  "lastModified": "2025-12-31T19:20:22.000Z",
  "author": "AaryanK",
  "downloads": 128,
  "likes": 10,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "gguf",
  "siblings_count": 8
}