---

license: apache-2.0

library_name: llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• llama-cpp
• imatrix
• bonsai

base_model:

• prism-ml/Bonsai-27B-unpacked

---

Bonsai-27B — Standard GGUF Ladder (Q3_K_M → Q8_0)

Imatrix GGUF quants of prism-ml's Bonsai-27B (Qwen3.6-27B backbone, 262K context, hybrid attention, vision), covering the quality tiers between the official QAT releases and F16.

Which repo should you use?

If you have ≤8GB: use prism-ml's official QAT quants, not these.

• Q1_0 (3.8GB) — 89.5% of F16 quality at 1.125 bits
• Ternary (7.2GB) — 95% of F16

Those are quantization-aware-trained with custom kernels — at their sizes, they beat anything post-training quantization can produce, including anything in this repo.

This repo covers the gap above them: the standard ladder for 12–32GB setups where you want maximum quality per GB, quantized from prism-ml's own F16 GGUF with an importance matrix (63KB coding/reasoning calibration corpus).

Quants

| File | Size | Measured (GB10, 273GB/s) | + DSpark drafter |

|---|---|---|---|

| Q8_0 | 28.6 GB | 7.8 tok/s | — |

| Q6_K | 22.1 GB | 9.1 tok/s | — |

| Q5_K_M | 19.2 GB | 10.2 tok/s | — |

| Q4_K_M | 16.6 GB | 12.0 tok/s | 15.7 tok/s (+31%) |

| IQ4_XS | 15.1 GB | 13.7 tok/s | — |

| Q3_K_M | 13.3 GB | 13.5 tok/s | — |

All tiers individually smoke-tested (coherent code generation, chat template engages via --jinja). Q8_0 quantized without imatrix (unneeded at 8-bit); all others use the included Bonsai-27B.imatrix. Decode rates scale with your memory bandwidth — a 936GB/s GPU (RTX 3090-class) will run ~3× these numbers.

Both official mmproj files are included for vision (--mmproj Bonsai-27B-mmproj-Q8_0.gguf).

Speculative decoding with the official DSpark drafter (+31% measured)

prism-ml ships a DSpark drafter (1.8GB) trained against this exact target model — and it works with this repo's quants, verified:

| Config | tok/s | Draft acceptance |

|---|---|---|

| Q4_K_M, mainline llama.cpp | 12.0 | — |

| Q4_K_M + DSpark drafter, prism fork | 15.7 | 79% (242/305) |

Output at temperature 0 is byte-identical with and without the drafter (speculative decoding is verify-always — the drafter can only affect speed, never quality).

Requires PrismML-Eng's llama.cpp fork (branch prism) — the dspark drafter architecture is not in mainline llama.cpp (yet; their pr/* branches suggest upstreaming is underway). Build and run:

git clone --depth 1 --branch prism https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server

./build/bin/llama-server \
  -m Bonsai-27B-Q4_K_M.gguf \
  -md Bonsai-27B-dspark-Q4_1.gguf \
  --spec-type draft-dspark --spec-draft-n-max 4 \
  -ngl 999 -ngld 999 -fa on -c 16384 -np 1 --jinja

Gotchas (from prism-ml's SPECULATIVE.md + our testing):

• --spec-draft-n-max must be exactly 4 (the drafter's block size) — other values crash at the first draft round
• Speculative mode disables cross-request prompt caching and concurrency (-np 1) — best for single-user/agent use
• Use -c 16384+ and a generous max_tokens — the model regularly thinks 1.5–2K tokens before the visible answer
• CUDA only for now (their Metal spec path is still being optimized)
• Expect bigger gains on higher-bandwidth GPUs (prism-ml measured ~1.9× on datacenter CUDA; our +31% is on a bandwidth-bound GB10)

Speed hunt: fastest Bonsai-27B config on a DGX Spark (GB10)

We benchmarked every acceleration path available for this model on GB10 (2026-07-14). Full matrix — 3-pass medians, coherence-gated, temp 0:

| Config | tok/s | Notes |

|---|---:|---|

| prism-ml Q1_0 QAT (3.5 GiB), prism fork | 43.7 | 🏆 fastest — llama-bench: tg128 44.14, pp512 1003 |

| prism-ml Ternary Q2_0 QAT (7.2 GB) | 29.3 | higher quality (95% of F16) |

| This repo Q4_K_M + DSpark drafter | 15.7 | +31% over base, 79% acceptance |

| This repo IQ4_XS | 13.7 | fastest PTQ tier without the fork |

| This repo Q4_K_M | 12.0 | baseline |

Finding — 1-bit QAT makes speculative decoding counterproductive: against the Q1_0 target, every drafting scheme failed to beat base speed on this hardware: the official DSpark drafter halved throughput (21.6 tok/s at 78% acceptance), a Q2_K requant of it was no better, Bonsai-8B-Q1_0 as a draft-simple drafter was an exact wash (43.9), and ngram speculation got 6% acceptance. At 3.5 GiB of weights, decode is no longer bandwidth-bound — there's nothing left for a drafter to amortize. KV-cache quantization and batch tuning were also flat. prism-ml's experimental megakernel/rmsnorm-qmv-fuse branch measured parity (42.8).

Practical guidance: on unified-memory hardware (GB10, Apple Silicon, Strix Halo), run the QAT Q1_0/Q2_0 plain — save the drafter for the PTQ tiers in this repo, where it genuinely pays (+31% on Q4_K_M). On high-bandwidth discrete GPUs the tradeoff may differ — measure via timings.draft_n_accepted.

Provenance

• Source: prism-ml/Bonsai-27B-gguf F16 (53.8GB) — quantized directly from their official F16 GGUF, no re-conversion
• llama.cpp: mainline, commit cecbf5fb0 (quantization + baseline numbers); PrismML fork 62061f91 branch prism (drafter numbers)
• imatrix: 63KB ChatML coding/debugging/reasoning corpus, 4096 ctx
• LICENSE/NOTICE carried over from the upstream repo (Apache-2.0)
• Quantized on NVIDIA DGX Spark (GB10, aarch64) — day-zero release, ~2.5h after the upstream drop

Report issues in the community tab — smoke-test failures, incoherence, or numbers that don't reproduce. Community benchmark reports welcome (include hardware, backend, and full launch command).