Ternary-Bonsai-27B (GGUF) – Run (Q2_0) Under 2 Minutes 🚀

This repository provides optimized pre-built binaries and configurations to get the Ternary-Bonsai-27B model up and running on Google Colab or similar GPU environments with acceleration in roughly 2 minutes when using the Q2_0 variant.

---

⚡ Quick Start

🔧 Setup (run once)

!pip install -q huggingface_hub

import os
from huggingface_hub import hf_hub_download

# --- CONFIGURATION ---
repo_id = "IamPradeep/Ternary-Bonsai-27B-GGUF-Colab-Prebuilt-GPU"

print("1. Downloading pre-compiled binary package...")
bin_zip_path = hf_hub_download(repo_id=repo_id, filename="llama_bin.tar.gz")

# Extract the binaries into a folder called 'llama_bin'
!mkdir -p ./llama_bin
!tar -xzvf {bin_zip_path} -C ./llama_bin

# Grant executable permissions to the binary
!chmod +x ./llama_bin/llama-cli

print("\n2. Downloading model (this takes ~1–2 minutes for 7.17 GB)...")
model_path = hf_hub_download(repo_id=repo_id, filename="Ternary-Bonsai-27B-Q2_0.gguf")

# Add the extracted folder to LD_LIBRARY_PATH for dynamic CUDA libraries
os.environ["LD_LIBRARY_PATH"] = f"./llama_bin:{os.environ.get('LD_LIBRARY_PATH', '')}"

---

🤖 Text Inference

# Run text inference!
system_prompt = "You are a helpful AI assistant."
prompt = "Explain quantum computing in simple terms."

print(f"\n--- Running inference with Ternary-Bonsai-27B ---\n")

!./llama_bin/llama-cli \
  -m "{model_path}" \
  -ngl 99 \
  -sys "{system_prompt}" \
  -p "{prompt}" \
  --temp 0.7 \
  --top-p 0.95 \
  -n 2048

---

👁️ Vision / Multimodal Inference (Live Upload)

You can also run vision tasks by downloading one of the vision projectors (mmproj) and uploading an image directly inside Google Colab:

from PIL import Image
from google.colab import files

# 1. Download Vision Projector (~629 MB)
print("Downloading Vision Projector...")
mmproj_path = hf_hub_download(repo_id=repo_id, filename="Ternary-Bonsai-27B-mmproj-Q8_0.gguf")

# 2. Live Image Upload
print("\n" + "="*50)
print("📸 PLEASE UPLOAD AN IMAGE FROM YOUR DEVICE:")
print("="*50)

uploaded = files.upload()

if not uploaded:
    print("\n❌ No file was uploaded!")
else:
    # Save the uploaded file path
    image_filename = list(uploaded.keys())[0]
    image_path = f"./{image_filename}"
    
    print(f"\n✅ Uploaded successfully: {image_filename}")
    display(Image.open(image_path))

    # 3. Run Multimodal Inference
    system_prompt = "You are an expert AI vision assistant. Provide clear, detailed, and accurate descriptions of images."
    prompt = "Describe what you see in this image in detail."

    print(f"\n🤖 Running Vision Inference on {image_filename}...\n")

    !./llama_bin/llama-cli \
      -m "{model_path}" \
      --mmproj "{mmproj_path}" \
      --image "{image_path}" \
      -sys "{system_prompt}" \
      -p "{prompt}" \
      -ngl 99 \
      --temp 0.2 \
      -n 2048

---

💡 Notes

• Model Download Speed: The default Ternary-Bonsai-27B-Q2_0.gguf file is ~7.17 GB. Downloading it inside Colab typically takes around 1 to 2 minutes depending on network throughput.
• Other Variants Available: This repository hosts alternative quantizations and vision projector files:
• Language Models:
• Ternary-Bonsai-27B-F16.gguf (53.8 GB)
• Ternary-Bonsai-27B-Q2_g64.gguf (7.59 GB)
• Ternary-Bonsai-27B-dspark-bf16.gguf (7.29 GB)
• Ternary-Bonsai-27B-PQ2_0.gguf (7.17 GB)
• Ternary-Bonsai-27B-Q2_0.gguf (7.17 GB)
• Ternary-Bonsai-27B-dspark-Q4_1.gguf (1.95 GB)

• Vision Projectors (mmproj):
• Ternary-Bonsai-27B-mmproj-BF16.gguf (931 MB)
• Ternary-Bonsai-27B-mmproj-Q8_0.gguf (629 MB)

Simply adjust the filename parameter in hf_hub_download to load your preferred variant.

• Architecture: Uses 1.58-bit ternary quantization (representing weights as $\{-1, 0, +1\}$), delivering high reasoning capability while keeping GPU VRAM requirements minimal.

---

🤝 Acknowledgments

Special thanks to PrismML, the original architecture creators behind the revolutionary 1-bit and ternary (1.58-bit) Bonsai model families. Their custom quantization methods allow this 27B parameter model to deliver rapid local GPU inference inside a compact ~7.2 GB footprint.