# Bonsai 2 27B benchmark scores are comparable; its throughput numbers are not Ternary Bonsai 2 27B's benchmark scores share identical H100 conditions, but its throughput table spans eight GPUs, two Apple chips, two packing formats… Canonical page: https://makerportal.ai/blog/on-device-ai-hf-prism-ml-ternary-bonsai-2-27b-gguf Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-09-18 Section: Comparison / On-Device AI · 8 min read Tags: on-device-ai, local-llm --- The benchmark scores in the README are measured under identical conditions (H100, vLLM, thinking mode) and are directly comparable to each other. The throughput numbers in the same document are not: they span eight different GPUs and two Apple Silicon chips, two packing formats, and a power envelope that ranges from 27.5 W to 455 W. Placing them in a single table without those columns is how a reader ends up treating "47 tok/s on a laptop" and "130 tok/s on a desktop" as the same kind of number. ## What it is Ternary Bonsai 2 27B is a ternary-quantized GGUF file derived from Qwen3.8-27B, distributed for llama.cpp with custom low-bit kernels. It targets engineers who need 27B-parameter reasoning on hardware with 8–32 GB of unified or dedicated memory. The weights use a {−1, 0, +1} representation with FP16 group-wise scaling (group size 128) and a blockwise Hadamard rotation (block 1024) folded into the stored weights. ## The benchmark figures prism-ml reports these were all measured on an NVIDIA H100 using EvalScope + vLLM in thinking mode. The conditions are identical across rows. | Model | bpw | Size | Avg score | % of FP16 | |-------|-----|------|-----------|-----------| | Qwen3.8-27B FP16 | 16.0 | 54 GB | 86.32 | 100% | | Qwen3.8-27B UD-Q4_K_XL | 5.2 | 17.6 GB | 85.18 | 98.7% | | **Bonsai 2 27B** | **1.72** | **5.9 GB** | **84.78** | **98.2%** | | Qwen3.8-27B IQ2_XXS | 2.8 | 9.4 GB | 72.59 | 84.1% | The category-level breakdown (same conditions) shows where the 98.2% average comes from: | Category | FP16 | Bonsai 2 | Δ | |----------|------|----------|---| | Math | 97.06 | 96.57 | −0.49 | | Coding | 89.07 | 89.42 | +0.35 | | Instruction following | 81.25 | 82.66 | +1.41 | | Agentic / tool calling | 76.74 | 74.92 | −1.82 | | Vision | 71.36 | 66.19 | −5.17 | | Knowledge & reasoning | 85.55 | 79.86 | −5.69 |

Reported by prism-ml/Ternary-Bonsai-2-27B-g… bits/weight· scroll →

Conditions around the comparison

scroll →

262K tokens
context length: 1.72 bits/weight
true for the model as a whole: 5.95 GB
PTQ1_0 deployed size: ~54 GB
float16 baseline size

Figure: values as prism-ml/Ternary-Bonsai-2-27B-gguf reports them, drawn from quantities extracted and quote-verified for this post. Bars are proportional within their shared unit; condition cells are exact values and are not scaled against one another. The studio has not re-measured them.

## The throughput figures prism-ml reports the following. The NVIDIA rows use the TG128 kernel; the Apple rows use Metal. The packing format varies by row. Board power is stated as a group range for NVIDIA cards and as a per-chip rail measurement for the M5 Pro. | Hardware | Packing | Decode (tok/s) | PP512 (tok/s) | J/tok | Power (as stated) | |----------|---------|----------------|---------------|-------|-------------------| | RTX 5090 (32 GB) | PQ2_0 | 129.9 | 3893 | 1.95 | 300–455 W (group) | | RTX 5090 (32 GB) | PTQ1_0 | 120.5 | 1805 | 2.15 | 300–455 W (group) | | RTX PRO 6000 Blackwell | PQ2_0 | 124.8 | 4020 | 2.49 | 300–455 W (group) | | H100 SXM (80 GB) | PQ2_0 | 113.9 | 2830 | 2.69 | 300–455 W (group) | | H100 SXM (80 GB) | PTQ1_0 | 86.9 | 1237 | 3.18 | 300–455 W (group) | | RTX 6000 Ada (48 GB) | PQ2_0 | 82.8 | 2431 | 2.51 | 300–455 W (group) | | RTX 6000 Ada (48 GB) | PTQ1_0 | 90.4 | 1657 | 2.49 | 300–455 W (group) | | RTX 4090 (24 GB) | PQ2_0 | 81.2 | 3124 | 2.99 | 300–455 W (group) | | RTX 4090 (24 GB) | PTQ1_0 | 91.1 | 1645 | 2.58 | 300–455 W (group) | | L40S (48 GB) | PQ2_0 | 74.4 | 2868 | 3.24 | 300–455 W (group) | | L40S (48 GB) | PTQ1_0 | 81.8 | 1543 | 2.82 | 300–455 W (group) | | A100 SXM (80 GB) | PQ2_0 | 73.9 | 1328 | 3.43 | 300–455 W (group) | | A100 SXM (80 GB) | PTQ1_0 | 54.7 | 706 | 4.28 | 300–455 W (group) | | L4 (24 GB, 72 W) | PQ2_0 | 29.8 | 777 | 2.42 | 300–455 W (group) | | L4 (24 GB, 72 W) | PTQ1_0 | 32.1 | 467 | 2.25 | 300–455 W (group) | | Apple M5 Pro (laptop) | PQ2_0 | 28.1 | 387 | — | 27.5 W GPU rail | | Apple M5 Max (laptop) | TG128 | 47.0 | 765 | — | not stated | | Apple M4 Pro (laptop) | TG128 | 18.0 | 125 | — | not stated | The M5 Pro appears in two sections of the README: the cross-platform table (28.1 tok/s, PQ2_0) and the Apple platforms section (28.7 tok/s, 7.2 GB, PP512 393). The 7.2 GB footprint is consistent with PQ2_0 at 7.21 GB. prism-ml states the M5 Pro figure "swings ~4% with background load," which accounts for the 28.1 vs 28.7 spread. ## Where the comparison holds and where it stops **Holds:** The four benchmark averages are directly comparable. Same hardware, same inference stack, same decoding mode, same scoring pipeline. The category breakdown is internally consistent with the same conditions. **Stops holding — bits per weight:** The benchmark table lists Bonsai 2 at 1.72 bpw. That is the "true" bits-per-weight counting all tensors, including the 26.2M parameters (0.0976% of the language model) held above ternary. The actual downloadable files are PTQ1_0 at 1.75 bpw (5.95 GB) or PQ2_0 at 2.13 bpw (7.21 GB). The 1.72 figure corresponds to the ideal 5.8 GB, which is not a shipped file. Meanwhile, IQ2_XXS is listed at 2.8 bpw and UD-Q4_K_XL at 5.2 bpw — these are the effective bits-per-weight of their actual deployed files. The table therefore compares Bonsai's theoretical minimum against competitors' actual deployed sizes. If you download PQ2_0, you are at 2.13 bpw, which narrows the gap to IQ2_XXS's 2.8 bpw considerably. **Stops holding — throughput across hardware:** The RTX 5090 at 129.9 tok/s and the M5 Pro at 28.1 tok/s are not a "desktop vs laptop" comparison in any actionable sense. The 5090 draws from a 300–455 W board power envelope (per-card TDP is not broken out in the README; the range covers the group). The M5 Pro GPU rail draws 27.5 W. That is a power ratio of roughly 11:1 to 16:1. The J/tok column makes the NVIDIA side visible (1.95 J/tok on the 5090) but the energy column is empty for all Apple rows, so a per-watt comparison cannot be constructed from the data as published. **Stops holding — the 98.2% as a single number:** The average is pulled up by math and coding, where Bonsai 2 is within 0.5 points of or above FP16. Vision drops 5.17 points and knowledge & reasoning drops 5.69 points. If your workload is document understanding or open-domain QA, the relevant figure is 66.19 or 79.86, not 84.78. ## What cannot be concluded The 47 tok/s figure for the M5 Max is a decode throughput at an unspecified context length. The PP512 figure for the same chip is 765 tok/s, which is a prompt-processing measurement at 512 tokens. These are different operations with different bottlenecks. The README does not state the context length at which the 47 tok/s decode was measured, so it cannot be extrapolated to a 100K-token conversation. The 262K context claim is an architectural maximum inherited from the base model; no measurement at that context length is reported in the README. ## Who this changes a decision for An engineer currently running IQ2_XXS or a similar sub-4-bit build of a 27B model on a laptop with 16–32 GB unified memory, whose workload is weighted toward math, coding, or structured instruction following. For those categories, Bonsai 2 PTQ1_0 at 5.95 GB delivers scores at or above FP16 while using roughly 63% of the memory of IQ2_XXS (5.95 / 9.4 ≈ 0.63; the README states "less than two-thirds of its size"). The practical change on Monday: if your eval suite is math/coding-heavy, swap the model file and expect equal or better quality at lower memory pressure. If your eval suite includes vision or open-domain knowledge retrieval, the 5-point drops in those categories may be disqualifying and you should benchmark before committing. If your workload is vision-centric, nothing changes: the 66.19 vision score is 5.17 points below FP16's 71.36 — a 7.2% relative drop, and no throughput gain compensates for that in a production setting. If you want to verify the memory footprint for your specific GPU and context length before pulling the file, the [LLM GPU Memory Calculator](/lab/llm-vram-kvcache-calculator) estimates VRAM at different quantization levels. The same pattern of "file sizes sit next to each other but the performance claims do not share conditions" showed up in our [CoreML-Models note](/blog/on-device-ai-github-286898814) — the conditions column is where the actual information lives. --- *Method: this note was drafted by qwen/qwen3.8-27b from a single source — the model card for [prism-ml/Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf). Before publication an automated gate re-checked every extracted claim against the source document (86 claim(s) and 120 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (11 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run prism-ml/Ternary-Bonsai-2-27B-gguf's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Is the 1.72 bpw figure the size of the file I download? No. 1.72 bpw corresponds to the ideal 5.8 GB. The PTQ1_0 file is 1.75 bpw at 5.95 GB; the PQ2_0 file is 2.13 bpw at 7.21 GB. The 26.2M parameters held above ternary (the recurrent state path of the linear-attention layers plus normalization weights) are what push the ideal toward the actual file sizes. ### Can I compare the 47 tok/s M5 Max number to the 129.9 tok/s RTX 5090 number? Not meaningfully. The 5090 operates within a 300–455 W board power envelope (per-card TDP not broken out in the README). The M5 Max power draw is not stated at all. The M5 Pro, for which power is reported, draws 27.5 W on the GPU rail. Without a per-watt normalization for the M5 Max, the raw tok/s ratio of 2.76× does not represent a fair hardware comparison. ### Does the 98.2% apply to all task types equally? No. Math retains 99.5% of FP16 (96.57 vs 97.06), coding exceeds it (89.42 vs 89.07), but vision retains 92.8% (66.19 vs 71.36) and knowledge & reasoning retains 93.3% (79.86 vs 85.55). The 98.2% is an average across 14 benchmarks and is weighted toward the categories where the model performs closest to FP16. ### Can the benchmark scores be verified independently? The README states the evaluation used EvalScope + vLLM on H100 in thinking mode with specified generation parameters (temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0, xhigh reasoning effort). The 14-benchmark suite includes AIME26, LiveCodeBench, MMLU-Redux, BFCL v3, and MMMU-Pro among others. The studio has not reproduced these results. Individual per-benchmark scores are partially published (AIME26, LiveCodeBench, MMLU-Redux are given for Bonsai 2 and IQ2_XXS) but the full 14-benchmark breakdown is not.