---

quantized_by: vinpix

base_model: prism-ml/Ternary-Bonsai-8B-gguf

license: apache-2.0

language:

- en

library_name: gguf

tags:

- gguf

- llama.cpp

- ternary

- 1.58-bit

- bitnet

- cpu

- qwen3

pipeline_tag: text-generation

---

<div align="center">

🌳 Ternary-Bonsai-8B · TQ2_0 GGUF

1.58-bit ternary 8B that runs on stock llama.cpp — no fork required

<p>

<img src="https://img.shields.io/badge/format-GGUF-1f6feb?style=for-the-badge&logo=huggingface&logoColor=white" alt="GGUF">

<img src="https://img.shields.io/badge/precision-ternary%20%7B%E2%88%921%2C0%2C%2B1%7D-8957e5?style=for-the-badge" alt="ternary">

<img src="https://img.shields.io/badge/body-2.06%20bpw-2ea043?style=for-the-badge" alt="2.06 bpw">

<img src="https://img.shields.io/badge/size-2.33%20GiB-e3b341?style=for-the-badge" alt="2.33 GiB">

</p>

<p>

<img src="https://img.shields.io/badge/llama.cpp-mainline-000000?style=flat-square&logo=cplusplus" alt="llama.cpp">

<img src="https://img.shields.io/badge/LM%20Studio-supported-4b32c3?style=flat-square" alt="LM Studio">

<img src="https://img.shields.io/badge/Ollama-supported-000000?style=flat-square&logo=ollama" alt="Ollama">

<img src="https://img.shields.io/badge/context-64K-0969da?style=flat-square" alt="64K context">

<img src="https://img.shields.io/badge/license-Apache--2.0-750014?style=flat-square" alt="Apache-2.0">

</p>

_{A portable GGUF repack of <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf"><b>PrismML Ternary-Bonsai-8B</b></a> · quantized by <b>vinpix</b>}

</div>

---

<div align="center">

> 8B quality. 2.3 GB on disk. Runs on the llama.cpp you already have.

</div>

Ternary-Bonsai-8B is an 8B Qwen3-architecture model whose linear weights are ternary — every weight is one of {−1, 0, +1} with a shared FP16 scale per group. This repack stores that ternary body in the mainline TQ2_0 type so it loads on any current llama.cpp, LM Studio or Ollama with no fork and no special flags.

<table>

<tr>

<td>⚡</td><td><b>Tiny footprint</b><br>_{2.33 GiB — fits in RAM on almost anything}</td>

<td>🔌</td><td><b>Zero setup</b><br>_{Stock llama.cpp / LM Studio / Ollama}</td>

</tr>

<tr>

<td>🎯</td><td><b>Lossless body repack</b><br>_{Same {−1,0,+1} weights, just re-packed}</td>

<td>📏</td><td><b>64K native context</b><br>_{No RoPE scaling tricks}</td>

</tr>

</table>

> [!NOTE]

> This repository redistributes and repacks the model. All model capabilities come from PrismML. See the original model card for architecture, training and evaluations.

---

📦 Files

<div align="center">

| File | Body | Embed / Output | Body bpw | Size |

|:---|:---:|:---:|:---:|:---:|

| Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf | TQ2_0 ternary | Q4_0 | 2.06 | ~2.33 GiB |

</div>

Everything except the token-embedding and output (LM-head) tensors is stored as native ternary TQ2_0. Those two tensors are not ternary in the source model — they hold higher-precision information — so they are kept in Q4_0 to preserve it at a reasonable size.

---

⬇️ Download

pip install -U "huggingface_hub[cli]"

hf download vinpix/Bonsai-8B-TQ-GGUF \
  --include "Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf" \
  --local-dir ./

---

🚀 Run

<details open>

<summary><b>llama.cpp</b></summary>

llama-cli -m Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf \
  -p "The capital of France is" \
  -n 256 --temp 0.5 --top-p 0.85 --top-k 20

OpenAI-compatible server:

llama-server -m Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf -c 4096

</details>

<details>

<summary><b>LM Studio / Ollama</b></summary>

Point them at the .gguf file directly. No fork or extra flags are required.

</details>

<details>

<summary><b>Recommended sampling & threads</b></summary>

| Setting | Value |

|:---|:---|

| temperature | 0.5 |

| top_p | 0.85 |

| top_k | 20 |

| threads (-t) | your physical core count |

More threads than physical cores usually does not help and can hurt.

</details>

<details>

<summary><b>Low-RAM tip — KV-cache quantization</b></summary>

Shrink the KV cache at runtime for long contexts (does not modify the weights):

llama-cli -m Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 -p "..."

q8_0 cache is close to lossless; q4_0 saves more memory at a small quality cost.

</details>

---

🔬 Format details

TQ2_0 packs four ternary weights per byte (2 bits each) with one scale per block. A weight is reconstructed as:

w = q_ternary × scale where q_ternary ∈ {−1, 0, +1}

Because the source weights are already ternary, the body is a re-packing of the same {−1, 0, +1} values — not a re-quantization of full-precision weights — so it carries no additional accuracy loss. The embedding and LM-head are the only non-ternary weights in the source and are stored here in Q4_0.

<details>

<summary><b>Reproduce these files</b></summary>

Produced from the PrismML F16 source with mainline llama.cpp:

llama-quantize --token-embedding-type q4_0 --output-tensor-type q4_0 \
  Ternary-Bonsai-8B-F16.gguf \
  Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf TQ2_0

</details>

---

ℹ️ Notes & limitations

• This is a re-quantization / re-pack, not a re-train. All capabilities are inherited from Ternary-Bonsai-8B.
• The body uses portable TQ2_0 (group size 256). The official PrismML Q2_0 (group size 128) is marginally more faithful to the original grouping but requires the PrismML fork; this repack trades that for running on stock llama.cpp.
• CPU inference of ternary models is memory-bandwidth bound — tokens/second depend on your system's memory bandwidth and core count, so throughput varies from machine to machine.
• Native context length is 65,536 tokens (RoPE base 1e6); no scaling applied.

---

<div align="center">

🙏 Credits

Model & all capabilities — Prism ML · Ternary-Bonsai-8B · whitepaper

Quantization / repack — vinpix

License — Apache-2.0 (inherited from the source model)

_{⭐ If this repack is useful, a like on the repo helps others find it.}

</div>