Bonsai 2 27B: Ternary Weights Meet the 8 GB GPU — and the Upstream Tax

On September 17, 2026, PrismML released Bonsai 2 27B, a ternary-weight compression of Qwen3.8-27B that puts a 27B-class reasoning model into a 5.9 GB footprint — small enough for an 8 GB consumer GPU or an everyday laptop, with the vendor reporting 98.2% of the full-precision model's benchmark average. The HN thread hit 523 points in under a day, and the GGUF repo pulled 400K+ downloads within roughly 40 hours of the repo appearing on Hugging Face. Numbers like that make this either the most important local-inference release of the year or a masterclass in benchmark-friendly framing — and the truth, as usual, is in the runtime details.

This is the second Bonsai generation. The first Bonsai 27B (July 14, 2026) proved a 27B multimodal model could be compressed onto a phone; Bonsai 2 moves to a stronger base, Qwen3.8-27B, and pushes aggregate retention from ~95% to 98.2% at the same 5.9 GB deployment point. It is Apache 2.0, supports text+image input, carries a 262K-token context window, and — this is the part that matters for anyone running inference infrastructure — it cannot run on stock llama.cpp. The whole stack rides on a vendor fork, and that decision carries operational consequences the announcement page does not fully spell out.

The Claim, and the Number That Actually Matters

The headline is a memory story: FP16 Qwen3.8-27B is ~54 GB of weights. Bonsai 2 27B is 5.9 GB. That is a 9.0–9.3x reduction, at 1.72–1.75 effective bits per weight, while the vendor reports the compressed model scoring 84.78 average across 14 thinking-mode benchmarks against the FP16 model's 86.32 — 98.2% retention. The comparable conventional quantizations of the same base model tell you why this landed on HN's front page:

*The vendor's materials use 84.78 for the 14-benchmark average and 83.9 for a separate six-category aggregate; both work out to 98.2% retention. Either way, the gap to UD-Q4_K_XL is under half a point at one-third the size, and the gap to IQ2_XXS is over twelve points at two-thirds the size.

The model card also defines an "intelligence density" metric — benchmark average per deployed gigabyte, D = -log2(1 - score/100) / size_GB. Bonsai 2 lands at 0.469/GB against 0.199 for IQ2_XXS and 0.157 for UD-Q4_K_XL. Density is a marketing metric until you fix a memory budget — then it is the only metric: per deployed GB, this build delivers roughly 2.3x the usable intelligence of the densest conventional quantization of the same model.

How Ternary + Hadamard Actually Works (and What It Costs You)

The representation is the interesting engineering. Each weight is stored as one of three values — {-1, 0, +1} — with one FP16 scale factor shared per group of 128 weights ("ternary g128"). A ternary value carries log₂3 ≈ 1.585 bits of information, so the format's information-theoretic floor is ~1.58 bits/weight; amortizing the 16-bit scales over 128 weights brings the effective storage to ~1.72 bits/weight, against 16 for FP16.

The piece that makes it accurate is the blockwise Hadamard rotation (block size 1024, fixed ±1 signs): each weight matrix is transformed by an orthogonal rotation before the ternary assignment, and the runtime applies the matching inverse transform to activations. The rotation is folded into the stored weights offline — no extra bits, no extra weight traffic — but it means the model file declares its rotation as metadata, and a runtime either applies the transform or refuses to load. That single design decision is the root of the entire operational story below.

FP16 weights          offline compression                runtime (llama.cpp fork)
──────────────       ────────────────────────           ──────────────────────────────
  16 bits/wt    ──►  Hadamard rotate (block 1024)  ──►  load packed trits (PTQ1_0/PQ2_0)
  ~54 GB              ternarize {-1,0,+1} g128          per-block: unpack trits
                      1 FP16 scale per 128 wts            matmul in rotated basis
                      fold rotation into weights           apply FWHT to activations
                      pack PTQ1_0 (dense, 1.75 bpw)          │
                      or PQ2_0 (2-bit slots, 2.13 bpw)       ▼
                   ──►  5.95 GB / 7.21 GB file        correct logits out the other end

Coverage is end-to-end across the language model: embeddings, attention projections, MLP projections, and the LM head all go ternary — no high-precision escape hatches behind a low-bit label, a direct dig at conventional builds whose "2-bit" label is really 2.8 bpw average. Exactly 26.2M parameters (0.0976% of the language model — the recurrent state path of the linear-attention layers plus normalization weights) stay in higher precision. The vision tower ships separately as an optional ~0.63 GB Q8_0 mmproj pack, loaded only when an image arrives; text-only serving never pays for it.

The architecture it rides on matters too: Qwen3.8-27B is a hybrid-attention model (~75% linear / ~25% full attention), which is what keeps a 262K context practical on a device — the KV cache costs ~64 KiB/token FP16 (~6.3 GiB at 100K), and with the optional 4-bit KV cache (BONSAI_KV4=1) that drops to ~18 KiB/token (~1.8 GiB at 100K). The base model does the heavy lifting on context economics; Bonsai's contribution is making the weights cheap. A thinking model, it serves with reasoning enabled by default and accepts --reasoning-budget to cap thinking tokens server-wide.

Real Throughput on Real Hardware

The GGUF model card ships measured llama-bench numbers (batch size 1, no vision tower) across the two packings, and the packing trade-off is a genuine fork in the road, not a versioning nicety:

Read the pattern: PTQ1_0 (dense trits) wins on Ada-class cards and the L4 — where memory bandwidth is the binding constraint and moving 17% less weight data per step pays — while PQ2_0 (2-bit slots) wins on Ampere, Hopper, and Blackwell, where batch-1 decode is limited by instruction throughput and launch overhead rather than bandwidth. Prompt processing is compute-bound and favors PQ2_0 everywhere. If you deploy this, you do not pick a packing once; you pick per hardware generation, and the vendor's own guidance says so.

Community numbers from the HN thread fill in the gaps the vendor table skips — the hardware people actually own:

Two operator-grade takeaways fall out of the community data. First, the "8 GB GPU" story is real but conditional: the weights fit, the KV cache needs quantization, and you are trading prompt-processing speed for the privilege. Second, the vendor throughput table is measured on the fork's kernels with no vision tower at depth 0 — your numbers will differ, mostly downward, the moment you enable the features you actually wanted (vision, long context, tools).

The Fork Tax: What "Runs on llama.cpp" Actually Means

Here is the part that should be in every platform team's evaluation notes. Bonsai 2 27B does not run on stock llama.cpp — and the failure mode is worse than a clean refusal:

For MLX users the story is better but not clean either: the MLX 2-bit pack runs on stock packages, no fork — but it declares model_type: prism_hadamard_qwen35 and requires the loader bundled in the repo's runtime/ directory. Ordinary MLX loaders skip the activation transform and the inverse embedding lookup, so they return wrong output rather than an error — the same silent-garbage trap in a different runtime. The loader is a few lines of Python (sys.path.insert plus load_vl_model), but your MLX-based pipeline will not pick it up by accident; it must be wired deliberately.

Deployment: Serving It for Real

The demo repo is the source of truth and is unusually well-built for a vendor demo — pinned binaries, per-platform build scripts, an environment-variable contract, and an AGENTS.md written for AI coding agents. The happy path on a Linux CUDA box:

# Option A: the demo repo (setup.sh handles binaries + weights + extras)
git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
./setup.sh                                  # downloads 7.8 GB PQ2_0 + mmproj by default
./scripts/start_llama_server.sh              # chat UI at http://localhost:8080
BONSAI_KV4=1 ./scripts/start_llama_server.sh --reasoning-budget 2048

# Option B: hand-rolled, for when the demo's defaults don't fit your serving stack
# 1. fork binary — grab the archive for your platform from the fork's releases page,
#    or build it yourself:
git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

# 2. weights — PQ2_0 for H100/A100/Blackwell, PTQ1_0 for Ada and memory-tight targets
hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir models

# 3. serve with the flags that matter
./build/bin/llama-server \
  -m models/Ternary-Bonsai-2-27B-PQ2_0.gguf \
  --port 8080 -ngl 99 -fa on -c 131072 \
  --reasoning-budget 2048 \
  --temp 1.0 --top-p 0.95 --top-k 20          # thinking-mode sampling, matches generation_config

# 4. tight-VRAM variant (8 GB class): quantize the KV cache, shrink batches
./build/bin/llama-cli -m models/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
  -ngl 99 -fa on -c 32768 -b 256 -ub 64 \
  -ctk q8_0 -ctv q8_0 \
  --temp 1.0 --top-p 0.95 --top-k 20 -p "your prompt" -n 256

Details the quickstart glosses over and that bite in production:

• Sampling defaults are baked into the GGUF metadata (general.sampling.*) and match the base model's generation_config.json: thinking mode wants temperature=1.0, top_p=0.95, top_k=20; instruct mode 0.7/0.80/20 plus presence_penalty=1.5. The benchmark numbers were produced with the thinking-mode settings — a client that overrides them with its own defaults is silently evaluating a different model.
• Reasoning effort is xhigh by default, low is not supported (selecting it behaves like xhigh), and medium is the documented balance point. On slow hardware, thinking tokens are the bulk of wall-clock latency; cap them explicitly or your latency SLOs are fiction.
• Context defaults are RAM-tiered (8K to 131K), not the full 262K — the launch scripts will not silently give you the training max. If you need 262K, say so explicitly and provision for the KV cache: ~6.3 GiB FP16 at 100K, ~1.8 GiB with 4-bit KV.
• Vision costs are pay-per-use: the ~0.63 GB mmproj pack loads only on image input, and BONSAI_MMPROJ_CPU=1 keeps it in system RAM to free ~0.9 GiB VRAM on tight cards, at the price of slower image prefill.
• BONSAI_HOST binds loopback by default. A non-loopback bind exposes an unauthenticated llama-server on your network — the README's own security note. Behind a gateway, add auth at the proxy; llama-server itself has none.

Tool calling is native OpenAI-style tool_calls with full round-trips, and the demo ships an MCP client — which is where a 27B-class local model gets genuinely interesting for platform teams: a model that can call tools, read screenshots, and reason over a repo, running on the laptop it was issued on, with data that never leaves the device.

Energy: The Number That Changes Fleet Math

The claim that deserves more attention than it got: on an RTX 4090, Bonsai 2 consumes 0.714 mWh/token — the vendor says 40% more energy-efficient than an 8B model at full precision. The throughput table makes the mechanism visible: the L4 (72 W TDP) decodes at 2.25 J/tok, an M5 Pro laptop draws 27.5 W on the GPU rail decoding at 28.1 tok/s, against 300–455 W board power for the big NVIDIA cards. Low-bit weights mean fewer bytes moved per token, and decode is bandwidth-bound — so the same reasoning capability lands at a fraction of the energy per token.

Where that changes fleet math: an L4 or a laptop GPU serving 27B-class reasoning at ~2.25–2.5 J/tok versus an H100 at 2.69 J/tok — and the L4 costs a fraction of an H100. For internal tooling endpoints (CI summarization, log triage, agent harnesses for developers, document Q&A over sensitive corpora) the quality bar is "good enough reasoning at minimal cost," and that is exactly the slot this model targets. It is not going to replace your frontier-API tier; it changes what the cheap tier can do.

Who Should Skip This

Skip if you need battle-tested serving. The runtime is a vendor fork, upstreaming is a series of PRs that has just begun (CPU FWHT merged Sep 18; GPU kernels pending), there is no speculative-decoding drafter, and the wider serving ecosystem — vLLM, SGLang, TensorRT-LLM — has no published support for these custom ggml types. If your serving stack is vLLM-with-Helm and it must stay that way, this is not for you yet; watch the upstream PRs and revisit when mainline can run the files.

Skip if your workload is knowledge- or vision-heavy at the frontier. The retention gap is concentrated exactly there: knowledge & reasoning 79.86 vs 85.55, vision 66.19 vs 71.36. A model that is 98.2% of FP16 in aggregate is 93% on the categories your eval set should care about. Run your own eval before believing any retention number — vendor or ours.

Take it seriously if: you operate developer fleets and want credible local reasoning agents (coding, tool use, document analysis) without paying per token or shipping data off-device; you run edge or cost-constrained inference where 5.9 GB for 27B-class quality changes what hardware is eligible; or you are planning capacity for the post-quantization era and want a live case study in what production-grade sub-2-bit inference looks like. The 400K downloads in two days say the demand is real; the engineering says the technique is arriving, not the hype. The one honest caveat: "near-lossless" is a benchmark aggregate, not a guarantee on your distribution — and the day a vendor claims otherwise is the day you should stop reading their whitepapers.

Verification status: all claims above are sourced from PrismML's announcement (Sep 17, 2026), the GGUF and MLX model cards, the Bonsai-demo repository, and measured community reports from the HN thread. Platform Monkey has not independently benchmarked this model; the throughput and energy figures are the vendor's own llama-bench measurements and HN participants' reports, reproduced as reported. Upstream merge status was verified against ggml-org/llama.cpp PRs #27779 and #27461 at publication time.