Bonsai-27B-1bit (GGUF) – Run in Under 1 Minute 🚀

This repository provides optimized pre-built binaries and configurations to get the Bonsai-27B-1bit model up and running on Google Colab or similar GPU environments with acceleration in under a minute.

⚡ Quick Start

🔧 Setup (run once)

!pip install -q huggingface_hub

import os
from huggingface_hub import hf_hub_download

# --- CONFIGURATION ---
repo_id = "IamPradeep/Bonsai-27B-1bit-GGUF-Colab-Prebuilt-GPU"

print("1. Downloading pre-compiled binary...")
bin_zip_path = hf_hub_download(repo_id=repo_id, filename="llama_bin.tar.gz")

# Extract the binaries into a folder called 'llama_bin'
!mkdir -p ./llama_bin
!tar -xzvf {bin_zip_path} -C ./llama_bin

# Ensure Linux knows it is an executable
!chmod +x ./llama_bin/llama-cli

print("\n2. Downloading model (this takes ~30–60 seconds)...")
model_path = hf_hub_download(repo_id=repo_id, filename="Bonsai-27B-Q1_0.gguf")

# FIX: Update the system path via Python instead of Bash
os.environ["LD_LIBRARY_PATH"] = f"./llama_bin:{os.environ.get('LD_LIBRARY_PATH', '')}"

---

🤖 Run Inference

# 3. Run the model!
system_prompt = "You are a helpful AI assistant."
prompt = "Explain quantum computing in simple terms."

print(f"\n--- Running inference ---\n")

!./llama_bin/llama-cli \
  -m "{model_path}" \
  -ngl 99 \
  -sys "{system_prompt}" \
  -p "{prompt}" \
  --temp 0.7 \
  --top-p 0.95 \
  -n 2048

---

🤝 Acknowledgments

Special thanks to PrismML, the original architecture creators behind the revolutionary 1-bit and ternary (1.58-bit) Bonsai model families. Their custom quantization methods allow this massive 27B parameter model to fit inside an ultra-compact ~3.8 GB footprint while maintaining fast local inference on free-tier GPUs.