davidbpunkt/bonsai-4b-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "library_name": "llama.cpp",
    "pipeline_tag": "text-generation",
    "tags": [
      "1-bit",
      "gguf",
      "llama-cpp",
      "cuda",
      "metal",
      "on-device",
      "prismml",
      "bonsai"
    ],
    "frontmatter": {
      "license": "apache-2.0",
      "library_name": "llama.cpp",
      "pipeline_tag": "text-generation",
      "tags": [
        "1-bit",
        "gguf",
        "llama-cpp",
        "cuda",
        "metal",
        "on-device",
        "prismml",
        "bonsai"
      ]
    },
    "hero_image_url": "./assets/bonsai-logo.svg",
    "summary": "End-to-end 1-bit language model for llama.cpp (CUDA, Metal, CPU) > **14.1x** smaller than FP16 | **4.2x** faster on RTX 4090 | runs on any device",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nlibrary_name: llama.cpp\npipeline_tag: text-generation\ntags:\n  - 1-bit\n  - gguf\n  - llama-cpp\n  - cuda\n  - metal\n  - on-device\n  - prismml\n  - bonsai\n---\n\n<p align=\"center\">\n  <img src=\"./assets/bonsai-logo.svg\" width=\"280\" alt=\"Bonsai\">\n</p>\n\n<p align=\"center\">\n  <a href=\"https://prismml.com\"><b>Prism ML Website</b></a> &nbsp;|&nbsp;\n  <a href=\"https://github.com/PrismML-Eng/Bonsai-demo/blob/main/1-bit-bonsai-8b-whitepaper.pdf\"><b>Whitepaper</b></a> &nbsp;|&nbsp;\n  <a href=\"https://github.com/PrismML-Eng/Bonsai-demo\"><b>Demo &amp; Examples</b></a> &nbsp;|&nbsp;\n  <a href=\"https://colab.research.google.com/drive/1EzyAaQ2nwDv_1X0jaC5XiVC3ZREg9bdG?usp=sharing\"><b>Colab Notebook</b></a> &nbsp;|&nbsp;\n  <a href=\"https://discord.gg/prismml\"><b>Discord</b></a>\n</p>\n\n# Bonsai-4B-GGUF-1bit\n\nEnd-to-end 1-bit language model for llama.cpp (CUDA, Metal, CPU)\n\n> **14.1x** smaller than FP16 | **4.2x** faster on RTX 4090 | runs on any device\n\n## Highlights\n\n- Deployed footprint fits on virtually any device with a GPU\n- **End-to-end 1-bit weights** across embeddings, attention projections, MLP projections, and LM head\n- **GGUF Q1_0_g128** format for 1-bit packing of weights with shared scales for each group (group size 128).\n- **Cross-platform**: CUDA (RTX/datacenter), Metal (Mac), Swift (iPhone/iPad), Android\n- **MLX companion**: also available as [MLX 1-bit g128](https://huggingface.co/prism-ml/Bonsai-4B-mlx-1bit) for native Apple Silicon inference\n\n<p align=\"center\">\n  <img src=\"./assets/frontier.svg\" width=\"680\" alt=\"Frontier Efficiency\">\n</p>\n\n## Resources\n\n- **[Google Colab](https://colab.research.google.com/drive/1EzyAaQ2nwDv_1X0jaC5XiVC3ZREg9bdG?usp=sharing)** — try Bonsai in your browser, no setup required\n- **[Whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/1-bit-bonsai-8b-whitepaper.pdf)** — for more details on Bonsai, check out our whitepaper\n- **[Demo repo](https://github.com/PrismML-Eng/Bonsai-demo)** — comprehensive examples for serving, benchmarking, and integrating Bonsai\n- **[Discord](https://discord.gg/prismml)** — join the community for support, discussion, and updates\n- **1-bit kernels**: [llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) (CUDA + Metal) · [MLX fork](https://github.com/PrismML-Eng/mlx) (Apple Silicon) · [mlx-swift fork](https://github.com/PrismML-Eng/mlx-swift) (iOS/macOS)\n- **[Locally AI](https://locallyai.app/)** — we have partnered with Locally AI for iPhone support\n\n## Model Overview\n\n| Item           | Specification                                                          |\n| :------------- | :--------------------------------------------------------------------- |\n| Parameters     | 4.0B (~3.6B non-embedding)                                             |\n| Architecture   | Qwen3-4B dense: GQA (32 query / 8 KV heads), SwiGLU MLP, RoPE, RMSNorm |\n| Layers         | 36 Transformer decoder blocks                                          |\n| Context length | 32,768 tokens                                                          |\n| Vocab size     | 151,936                                                                |\n| Weight format  | GGUF Q1_0_g128                                                         |\n| Deployed size  | **0.57 GB** (14.2x smaller than FP16)                                  |\n| 1-bit coverage | Embeddings, attention projections, MLP projections, LM head            |\n| License        | Apache 2.0                                                             |\n\n## Quantization Format: Q1_0_g128\n\nEach weight is a single bit: `0` maps to `−scale`, `1` maps to `+scale`. Every group of 128 weights shares one FP16 scale factor.\n\nEffective bits per weight: **1.125** (1 sign bit + 16-bit scale amortized over 128 weights).\n\n\n### Memory Requirement\n\nParameter memory only (weights and scales loaded into memory):\n\n| Format             | Size        | Reduction | Ratio     |\n| :----------------- | ----------: | --------: | --------: |\n| FP16               | 8.04 GB     | —         | 1.0x      |\n| **GGUF Q1_0_g128** | **0.57 GB** | **93.0%** | **14.2x** |\n| MLX 1-bit g128     | 0.63 GB     | 92.2%     | 12.8x     |\n\nThe GGUF file on disk is 0.57 GB (~6.4 MB larger) because the format embeds the tokenizer, chat template, and model metadata alongside the weights.\n\n## Best Practices\n\n### Generation Parameters\n\n| Parameter          | Default | Suggested range |\n| :----------------- | :------ | :-------------- |\n| Temperature        | 0.5     | 0.5 -- 0.7      |\n| Top-k              | 20      | 20 -- 40        |\n| Top-p              | 0.9     | 0.85 -- 0.95    |\n| Repetition penalty | 1.0     |                 |\n| Presence penalty   | 0.0     |                 |\n\n### System Prompt\n\nYou can use a simple system prompt such as:\n\n```\nYou are a helpful assistant\n```\n\n## Quickstart\n\n### llama.cpp (CUDA)\n\n```bash\n# Clone the PrismML fork of llama.cpp (includes Q1_0_g128 kernels)\ngit clone https://github.com/PrismML-Eng/llama.cpp\ncd llama.cpp\n\n# Build with CUDA support\ncmake -B build -DGGML_CUDA=ON && cmake --build build -j\n\n# Run inference\n./build/bin/llama-cli \\\n    -m Bonsai-4B-Q1_0_g128.gguf \\\n    -p \"Explain quantum computing in simple terms.\" \\\n    -n 256 \\\n    --temp 0.5 \\\n    --top-p 0.85 \\\n    --top-k 20 \\\n    -ngl 99\n```\n\n### llama.cpp (Metal / macOS)\n\n```bash\n# Clone the PrismML fork of llama.cpp (includes Q1_0_g128 kernels)\ngit clone https://github.com/PrismML-Eng/llama.cpp\ncd llama.cpp\n\n# Build with Metal support (default on macOS)\ncmake -B build && cmake --build build -j\n\n# Run inference\n./build/bin/llama-cli \\\n    -m Bonsai-4B-Q1_0_g128.gguf \\\n    -p \"Explain quantum computing in simple terms.\" \\\n    -n 256 \\\n    --temp 0.5 \\\n    --top-p 0.85 \\\n    --top-k 20 \\\n    -ngl 99\n```\n\n### llama.cpp Server\n\n```bash\n./build/bin/llama-server \\\n    -m Bonsai-4B-Q1_0_g128.gguf \\\n    --host 0.0.0.0 \\\n    --port 8080 \\\n    -ngl 99\n```\n\nOpen the web UI at [http://127.0.0.1:8080](http://127.0.0.1:8080), or see our [llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp) for more examples.\n\n## Cross-Platform Throughput\n\n| Platform     | Backend         | TG128 (tok/s) | FP16 TG (tok/s) | TG vs FP16 | PP512 (tok/s) | FP16 PP512 (tok/s) |\n| :----------- | :-------------- | ------------: | --------------: | ---------: | ------------: | -----------------: |\n| RTX 4090     | llama.cpp CUDA  | 440           | 105             | **4.2x**   | 18,135        | 16,310             |\n| M4 Pro 48 GB | llama.cpp Metal | 136           | 29              | **4.7x**   | 915           | 915                |\n\n## Citation\n\nIf you use 1-bit Bonsai 4B, please cite:\n\n```bibtex\n@techreport{bonsai,\n    title   = {Bonsai: End-to-End 1-bit Language Model Deployment\n               Across Apple, GPU, and Mobile Runtimes},\n    author  = {Prism ML},\n    year    = {2026},\n    month   = {March},\n    url     = {https://prismml.com}\n}\n```\n\n## Contact\n\nFor questions, feedback, or collaboration inquiries: **contact@prismml.com**\n",
    "related_quantizations": []
  },
  "tags": [
    "llama.cpp",
    "gguf",
    "1-bit",
    "llama-cpp",
    "cuda",
    "metal",
    "on-device",
    "prismml",
    "bonsai",
    "text-generation",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 224,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-03T07:40:24.000Z",
  "created_at": "2026-04-03T07:40:24.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "llama.cpp"
}

{
  "_id": "69cf6ee8efcea0dd81d56eab",
  "id": "DavidBPunkt/Bonsai-4B-gguf",
  "modelId": "DavidBPunkt/Bonsai-4B-gguf",
  "sha": "4aef055eb0c2922602132454928d7e17b8f7932f",
  "createdAt": "2026-04-03T07:40:24.000Z",
  "lastModified": "2026-04-03T07:40:24.000Z",
  "author": "DavidBPunkt",
  "downloads": 224,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "llama.cpp",
  "siblings_count": 8
}