Ternary Bonsai 2 on One RTX 2080 Ti: Qwen3.8-27B in 7.2 GB Scores 130 of 140

Preface

A vacuum storage bag shrinks a duvet to a quarter of its size. You pay for it later: every time you want the duvet, you open the bag and wait for it to puff back up.

Part 14 ran Qwen3.8-27B as Q8_0 across two modded 2080 Tis, with its built-in MTP draft head. After tuning it decoded at 66 tok/s and became this machine's main serving setup. This part takes the same model compressed to ternary weights, fits it on one card, and asks how much of the model survives. On my 140-question set, one question separates them: 130 vs 131.

What Ternary Bonsai 2 is: Qwen3.8-27B with nearly every weight at −1, 0 or +1

Ternary Bonsai 2 27B is a model from PrismML, derived from Qwen3.8-27B. Nearly every weight is −1, 0 or +1, and each group of 128 weights shares one FP16 scale. Three possible values carry log2(3) ≈ 1.58 bits of information per weight. Averaged over the whole model, it comes to 1.72 bits per weight. The linear-attention state path and the norm weights, about 0.1% of the language model, stay at higher precision; the 1.72 already counts them.

The weights are stored in a rotated basis. The rotation is a Hadamard transform: a fixed orthogonal matrix made of +1s and −1s, applied here in blocks of 1024. Because the weights were rotated, the runtime has to apply the matching transform to each layer's input activations before the multiply. Stock llama.cpp has no such step, so it rejects the two Bonsai formats, PQ2_0 and PTQ1_0, as unknown types. The model card also warns that stock llama.cpp loads Q2_0 files without complaint and outputs garbage. The rotation matters again in the speed section.

The same ternary weights ship in two packings. The 5.9 GB in PrismML's headline is PTQ1_0.

packing	how each trit is stored	bits/weight	file
PTQ1_0	packed densely	1.75	5.95 GB
PQ2_0	one per 2-bit slot	2.13	7.21 GB

A trit is one ternary digit. PQ2_0 wastes part of each 2-bit slot, and in exchange its trits are cheaper to unpack.

PrismML's claim: 98.2% of FP16 in 5.9 GB, where IQ2_XXS keeps 84.1%

These numbers come from the model card, not from my runs: H100, EvalScope with vLLM, thinking mode, averaged over 14 benchmarks. Compare the last column for IQ2_XXS and Bonsai 2, which sit in the same ~2-bit class.

variant	size	14-benchmark average	vs FP16
Qwen3.8-27B FP16	54 GB	86.32	100%
UD-Q4_K_XL	17.6 GB	85.18	98.7%
IQ2_XXS	9.4 GB	72.59	84.1%
Bonsai 2 27B	5.9 GB	84.78	98.2%

At about 2 bits, the conventional quant keeps 84.1%. Bonsai 2 keeps 98.2%, half a percentage point below the 17.6 GB UD-Q4_K_XL.

130 vs 131 out of 140: one card against the two-card Q8_0 setup

My eval set is 140 questions: 50 from GSM8K, 30 from MATH-500, 20 from MuSR and 40 from HumanEval+, drawn with a fixed seed. Both setups got the same questions and the same scoring. Thinking was on. Sampling followed Qwen's recommendation: temperature 1.0, top_p 0.95, top_k 20. Each question got an 8,192-token output cap, and requests ran one at a time.

Compare the scores against the card counts:

setup	cards	weights	score /140	median decode
Bonsai 2 PQ2_0, reasoning effort medium	1	7.2 GB	130	36.85 tok/s
Bonsai 2 PQ2_0, effort xhigh (fork default)	1	7.2 GB	116	35.38 tok/s
Qwen3.8-27B Q8_0 (abliterated) + MTP, llama.cpp	2	29 GB	131	66.04 tok/s

Per benchmark, medium-effort Bonsai vs the Q8_0 setup: MATH-500 30/30 on both, MuSR 15/20 against 13/20 with Bonsai ahead, HumanEval+ 36/40 against 38/40, GSM8K 49/50 against 50/50.

130/131 is 99.2% of the Q8_0 setup. That is a different yardstick from PrismML's 98.2%. Theirs compares against FP16 on 14 benchmarks on an H100. Mine compares against Q8_0 on 4 benchmarks and 140 questions.

Two caveats:

• I ran the 140 questions on the 7.2 GB PQ2_0, not the 5.9 GB PTQ1_0 in the headline. The ternary weights are the same, so capability should match, but on PTQ1_0 I only measured speed.
• This compares two setups, not two quantizations. The Q8_0 side also runs MTP across two cards, so the gap isn't purely quantization.

Why the smallest file decodes slowest: Bonsai uses 46% of the card's bandwidth, Q4_K uses 78%

These are llama-bench runs on the same fork and one 2080 Ti. tg128 is decode speed over 128 generated tokens; that's the column to watch.

weights	file	pp512	tg128	total VRAM
PTQ1_0	5.95 GB	435 tok/s	32.71 tok/s	6,670 MiB
PQ2_0	7.21 GB	691 tok/s	39.12 tok/s	7,818 MiB
Q4_K (same base model)	15.66 GiB	634 tok/s	28.68 tok/s	16,330 MiB

The VRAM readings include 346 MiB that ComfyUI was holding on that card.

The smallest packing is the slowest. PQ2_0 is 1.2 GB bigger than PTQ1_0 and decodes 19.6% faster. Against Q4_K, PQ2_0 decodes 36% faster in less than half the VRAM.

To see why, compute how much of the card's memory bandwidth decode actually uses. Each generated token reads every weight once, so decode tok/s × weight file size is the number of bytes read from VRAM per second. Divide that by the 2080 Ti's theoretical 616 GB/s (352-bit bus × 14 Gbps):

• Q4_K: 16.8 GB (15.66 GiB) × 28.68 = 482 GB/s, 78% of peak
• PQ2_0: 7.21 GB × 39.12 = 282 GB/s, 46%
• PTQ1_0: 5.95 GB × 32.71 = 195 GB/s, 32%

Q4_K runs close to the memory limit, so a smaller file means faster decode. Bonsai isn't limited by memory. Every token requires unpacking the trits and applying the Hadamard transform to the inputs, and that is arithmetic, not data movement. PTQ1_0 moves 17% less data than PQ2_0, but its denser packing costs more to unpack.

The model card says the same thing. PTQ1_0 is faster on Ada-generation cards and the L4, where decode is short on memory bandwidth. It loses to PQ2_0 on H100, A100 and Blackwell, where batch-1 decode is limited by instruction throughput and launch overhead. The 2080 Ti lands in the second group. Part 12 found that on this card, low-bit quantization saves VRAM but not necessarily time.

What you need: one 2080 Ti, PrismML's llama.cpp fork, and a 7.21 GB file

• One modded 2080 Ti 22G. The short-context peak is under 8 GB. By that math a stock 11 GB 2080 Ti should fit, but I didn't test one.
• PrismML's llama.cpp fork, PrismML-Eng/llama.cpp, branch prism. I used commit 01041db with CUDA 12.4. The build flag that matters is CMAKE_CUDA_ARCHITECTURES=75: the PQ2_0 and PTQ1_0 kernels compile for sm_75, the 2080 Ti's compute capability.
• The weights: Ternary-Bonsai-2-27B-PQ2_0.gguf, 7.21 GB, from prism-ml/Ternary-Bonsai-2-27B-gguf. On this card, pick PQ2_0 over PTQ1_0 for the bandwidth reasons above.

How to build and run it: clone, compile for sm_75, download, serve

Clone the fork and build the server and llama-bench for Turing:

git clone --branch prism https://github.com/PrismML-Eng/llama.cpp prismml-llama.cpp
cd prismml-llama.cpp
cmake -B build-cuda75 -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DCMAKE_BUILD_TYPE=Release
cmake --build build-cuda75 -j 20 --target llama-server llama-bench
./build-cuda75/bin/llama-server --version   # the fork's llama-bench has no --version; check the server instead

Download the PQ2_0 weights:

hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir ~/models/bonsai2

Start the server:

./build-cuda75/bin/llama-server \
  -m ~/models/bonsai2/Ternary-Bonsai-2-27B-PQ2_0.gguf \
  -ngl 99 -c 16384 -np 1 -fa on --jinja \
  --reasoning-effort medium \
  --host 0.0.0.0 --port 8080

This is the exact line that produced 130/140. I sent sampling parameters with each request: temperature 1.0, top_p 0.95, top_k 20, min_p 0. The model card says the GGUF metadata carries the same defaults.

Two blockers: stock llama.cpp, and an xhigh default that scores 116

Stock llama.cpp won't load PQ2_0 or PTQ1_0. It has no Hadamard runtime. Use the fork.

Reasoning effort defaults to xhigh. At that setting, 21 of 140 questions spent all 8192 tokens without answering, and the score was 116. With --reasoning-effort medium, one question hit the limit and the score was 130. The model card says low isn't supported and behaves close to xhigh, so use medium.

Not a blocker, but a time sink: a second card doesn't help. Tensor parallel (-sm tensor) hits an assertion at load, and layer split (-sm layer) adds only 6–7%. The deep dive explains why.

MTP adds 15% on code and 43% on math, with a community graft and a 15-line patch

MTP is Qwen3.8's built-in draft head. It guesses a few tokens ahead, the main model verifies them in one pass, and every correct guess comes free.

The official Bonsai 2 has no MTP head. Its GGUF lacks the original model's nextn tensors, and PrismML's download script says there is no draft model. A community build grafts MTP back: BoldingBuilds' abliterated (uncensored) PQ2_0-MTP.

Same card, temperature 0, median of 3 runs per workload:

config	code (tok/s)	math (tok/s)	prose (tok/s)
no speculation	38.9	38.2	37.8
MTP n=2	44.8	54.6	47.3
MTP n=3	40.3	53.1	41.8

n=2 is the fastest: +15% on code, +43% on math, +25% on prose. Drafting more tokens slows it down. The flags are --spec-type draft-mtp --spec-draft-n-max 2. The fork needs a 15-line patch before it will run the MTP graph; the patch is in the deep dive.

This build scores 128/140 at medium effort, two below the original and within noise (MATH-500 −3, MuSR +1). It's an uncensored (abliterated) build.

262K context on one 22 GB card: q8_0 KV cache, 17,312 MiB at load

Qwen3.8-27B has 64 layers. Of those, 48 use linear attention, which keeps a fixed-size state instead of a growing KV cache. Only the 16 full-attention layers store KV. That is what lets 262K tokens fit on one card:

./build-cuda75/bin/llama-server -m <PQ2_0-MTP.gguf> -ngl 99 \
  -c 262144 -ctk q8_0 -ctv q8_0 -np 1 -fa on --jinja \
  --spec-type draft-mtp --spec-draft-n-max 2

Load peak is 17,312 MiB. Below is a real 150K-token prompt on the abliterated PQ2_0-MTP.

	prefill	decode	peak VRAM
no speculation	315.5 tok/s	13.1 tok/s	17,336 MiB
MTP n=2	297.3 tok/s	21.1 tok/s	19,432 MiB

Reading 150K tokens takes about 8 minutes. An fp16 KV cache would need roughly 24 GB by my estimate, so q8_0 KV is mandatory here. Long context is where MTP pays off most: decode goes up 61%.

---

Deep dive: reasoning effort, the bandwidth math, DFlash2, and why -sm tensor asserts

You can skip this section. Everything above is enough to run the model; what follows is the full data, the verbatim errors and the commands behind each claim.

116 → 122 → 130: the fix was reasoning effort, not a bigger token budget

The first full run used the fork's default effort, xhigh, and scored 116/140. Of 24 failures, 21 ended with finish_reason=length: 15 on MuSR, 4 on HumanEval+, 2 on MATH-500. All 15 MuSR failures were still thinking when they hit the limit.

A bigger budget is the obvious fix, and it's what I tried first. I reran those 21 questions at 16384 tokens:

bench	rerun	rescued	still truncated at 16K	wrong
MATH-500	2	2	0	0
HumanEval+	4	2	2	0
MuSR	15	2	9	4

That brings the composite to 122/140. Doubling the budget rescued 6 of 21. The truncated reasoning wasn't looping: the last 5,000 characters had no repeated fragments. It was overthinking. On a two-choice murder-mystery question, it was checking the first letters of paragraphs for a hidden acrostic.

The model card recommends medium effort. A full 140-question rerun at medium scored 130/140, and truncations dropped from 21 to 1:

bench	N	xhigh/8192	medium/8192	Q8_0 setup
GSM8K	50	49	49	50
MATH-500	30	28	30	30
MuSR	20	5	15	13
HumanEval+	40	34	36	38
total	140	116	130	131

group	truncated	median output tokens	p90 output tokens	median decode (tok/s)
xhigh/8192	21	447	8192	35.38
medium/8192	1	512	2326	36.85
Q8_0 setup	0	522	2437	66.04

The medians are nearly equal. The p90 is what separates them: at xhigh, 10% of answers run into the wall.

How I checked the flag actually applied. The server log has no line for --reasoning-effort, and /props has no field for it. The fingerprint I used is the prompt token count. On the same question, gsm8k/895, xhigh gave prompt_n=135 and medium gave prompt_n=93. The flag feeds the chat template, and nothing else in the run changes how the prompt is rendered, so the missing 42 tokens can only come from it. Reasoning length on that question came out at 286, 209 and 270 characters across three groups. At temperature 1.0 that is sampling noise, not evidence.

Head-to-head, medium vs the Q8_0 setup:

• both right: 126
• only Bonsai right: 4 (musr/34, musr/212, musr/0, HumanEval/76)
• only Q8_0 right: 5 (gsm8k/1059, musr/20, HumanEval/145, HumanEval/97, HumanEval/151)
• both wrong: 5

Scoring. GSM8K takes the number after the last Answer:. MATH-500 takes the last complete \boxed{} and judges it with math_verify 0.9.0. MuSR takes the last option letter. HumanEval+ runs evalplus 0.3.1; both the base and plus tests must pass, with a 20 s outer timeout per problem. The sampling seed is random.Random(20260920). finish_reason=length counts as wrong, with no reruns.

Bandwidth usage: how I computed it, and a prefill number I got wrong

The method: each generated token reads all weights once, so decode tok/s × weight file size is the bytes read from VRAM per second. Divide by 616 GB/s (352-bit × 14 Gbps).

weights	file	tg128 (tok/s)	bytes read	of 616 GB/s
Q4_K	16.8 GB (15.66 GiB)	28.68	482 GB/s	78%
PQ2_0	7.21 GB	39.12	282 GB/s	46%
PTQ1_0	5.95 GB	32.71	195 GB/s	32%

Two more independent signals point the same way.

Speculation gets slower as it drafts more. DFlash2 (a separate draft model, covered in the next section) on Bonsai at n=3, 5 and 8 gave 46.3, 42.5 and 34.7 tok/s on code. In August I ran Qwen3.8-27B's own DFlash2 draft on a regular Q4 with the same sweep, and it climbed from 53.77 at n=2 to 59.73 at n=5 before leveling off. Speculation pays by verifying several tokens per read of the weights. When weight reads aren't the bottleneck, verifying n tokens just costs about n times the arithmetic.

A regular 4-bit quant on this card is still memory-bound. In August, same base model, same machine, pure decode: Q4 at 37.79 tok/s against Q8 at 27.80, +36%. There, smaller is faster.

A measurement mistake I made. I first tried prefill speed ÷ decode speed as a diagnostic and measured Bonsai's prefill at 21.2 tok/s, slower than its decode. I nearly wrote that up as proof. The probe sent the same 4,060-token prompt three times and took the median. Runs 2 and 3 hit the prompt cache and had only a few tokens left to process, so fixed overhead dragged their tok/s down, and the median landed on a cached run.

The counter-evidence was already there. Server startup plus all three requests finished within 20 s; a true 21.2 tok/s would need 191 s for one request. llama-bench measured pp512 at 691. The 150K-token prompt prefilled at 315.

If you measure prefill, use a different prompt on every run, or turn the cache off. And the ratio can't diagnose decode anyway. Prefill processes a large batch at once and is compute-heavy by nature. Bonsai's correct pp512 ÷ tg128 is 17.7×, which that diagnostic would read as memory-bound, the opposite of the other three signals.

DFlash2 on Bonsai 2 runs, but beats built-in MTP by only 3–8%

ProCreations published a DFlash2 draft model made for Bonsai 2, ProCreations/Ternary-Bonsai-2-27B-DFlash2, a 2.06 GB Q8_0 GGUF. The PrismML fork can't load it:

done_getting_tensors: wrong number of tensors; expected 81, got 58

The draft is the newer generation of DFlash2, with conv and selector modules. The fork's architecture table has no entries for DFLASH_ATTN_CONV_BASE/PROJ, DFLASH_FFN_CONV_BASE/PROJ or DFLASH_SELECTOR_HIDDEN/NEXT/PREV. I had also built the two upstream llama.cpp DFlash2 PRs; they fail earlier, on the target model, because upstream has no PQ2_0 support.

What works is the runtime source shipped in the ProCreations repo, runtime/prism-dflash2-source.tar.gz, already patched. Build it with:

cmake -S <src>/llama -B <src>/llama/build -G "Unix Makefiles" \
  -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build <src>/llama/build -j 12 --target llama-server llama-quantize

The repo lists CUDA 13.3, but find_package(CUDAToolkit) doesn't pin a version, and 12.4 builds fine. Without ninja installed, use the Unix Makefiles generator.

Same binary, same card, temperature 0, -c 16384, median of 3 per workload, on the abliterated version. All five rows ran on this binary, so its MTP n=2 row is a separate measurement from the MTP table in the first half.

config	code (tok/s)	math (tok/s)	prose (tok/s)	acceptance	peak VRAM
MTP n=2	42.8	50.4	45.4	60.9%	9.3 GB
DFlash2 n=3	46.3	52.4	46.8	63.3%	11.5 GB
DFlash2 n=5	42.5	50.2	44.1	50.3%	11.8 GB
DFlash2 n=8	34.7	46.9	36.8	39.9%	12.3 GB
original (not abliterated), DFlash2 n=5	47.3	49.0	40.9	50.7%	11.8 GB

The best DFlash2 setting, n=3, is only 3–8% ahead of built-in MTP. It costs 2.2 GB more VRAM and an extra draft file. Abliteration barely moves acceptance: 50.3% against 50.7% at n=5. Not worth it.

The same family of draft on the 29 GB FP8 model across two cards is a different story, because there the weights really are memory-bound. That is part 20.

Why -sm tensor asserts on two cards: 2560 ÷ 1024 = 2.5

With two cards and -sm tensor, the first error is:

llama_params_fit is not implemented for SPLIT_MODE_TENSOR

With --fit off, the real one appears:

ggml-backend-meta.cpp:1086: GGML_ASSERT(split_state.ne[j] % div == 0) failed

I first suspected the hybrid attention. But part 14's Q8_0 has the identical architecture to Bonsai 2's MTP version (both qwen35, 866 tensors, identical SSM parameters), and it splits fine. The only difference is Bonsai's extra GGUF metadata:

prism.hadamard.block_size    = 1024
prism.hadamard.transform     = normalized-sylvester-walsh-hadamard
prism.hadamard.axis          = input-last-dimension
prism.hadamard.weight_names  = 401 weights

The Hadamard transform acts on the input's last dimension in blocks of 1024. The hidden width is 5120. Split across two cards, that's 2560 per card, and 2560 ÷ 1024 = 2.5, so one block gets cut in half. split_state is tensor parallel's own bookkeeping, and it requires each card's slice to be divisible by the block size. This blocks MTP and DFlash2 alike; it doesn't depend on the speculation method.

-sm layer runs. It's a pipeline: one card computes the first half of the layers, the other computes the second half, and only one card works at a time. DFlash2 n=3 gave 49.0 / 55.4 / 50.1 tok/s and MTP n=2 gave 46.0 / 53.7 / 48.6 (code / math / prose), 6–7% over one card. The second card adds capacity, not compute.

The 15-line MTP patch for commit 01041db

At commit 01041db, the fork's main graph un-rotates the embeddings right after the lookup. The MTP graph has its own embedding lookup and skips that step, so llama_verify_hadamard_graph rejects the context. The patch goes in src/models/qwen35.cpp:

#include "llama-impl.h"   // llama_mul_mat_hadamard

// in graph_mtp, right after tok_embd = ggml_get_rows(...):
        // a Hadamard-latent embedding table stores rotated rows; restore the
        // primal basis right after the lookup: h = s * (H z). The main graph does
        // this in llm_graph_context::build_inp_embd(); this graph has its own lookup
        // and must do the same or llama_verify_hadamard_graph rejects the context.
        if (hadamard_inverses) {
            const auto it = hadamard_inverses->find(tok_embd_w);
            if (it != hadamard_inverses->end()) {
                tok_embd = llama_mul_mat_hadamard(ctx0, tok_embd, it->second.rot);
                if (it->second.signs) {
                    tok_embd = ggml_mul(ctx0, tok_embd, it->second.signs);
                }
            }
        }

After patching, --version still reports commit 01041db99. Keep your own record of which binary is patched.

Bonsai v1's dspark draft: 1.57× on arithmetic, 0.90× on Chinese

Bonsai 2 has no draft model. The previous Ternary-Bonsai-27B (v1) has one: a dspark drafter from the DFlash family, merged into upstream llama.cpp on 2026-07-28. It ships as bf16 at 7.29 GB. I dropped the shared tensors and quantized it to Q4_0, which brought it to 632 MB. Same server, same card, temperature 0, median of 3:

prompt	no spec (tok/s)	dspark (tok/s)	speedup	acceptance
EN quicksort	41.8	53.3	1.28×	0.57
zh-TW 300-character explanation	41.2	36.9	0.90×	0.31
EN car-chase arithmetic	40.7	63.9	1.57×	0.74

The official claim is "lossless 1.34×". My three-prompt average is 1.25×, and the Chinese prompt got slower (0.90×).

At temperature 0, the reasoning with and without speculation diverged around token 600, because batch verification's numerics flip an argmax. In llama.cpp's implementation, "lossless" does not mean byte-identical.

The draft-free ngram-simple made zero proposals on all three real prompts, for 1.00×. Reasoning prose has no repeated spans for it to grab.

Not tested yet

• A 140-question eval on the 5.95 GB PTQ1_0. It should match PQ2_0; I haven't measured it.
• A stock 11 GB 2080 Ti.
• Vision (mmproj).
• In an early smoke test, a zh-TW answer leaked one simplified-Chinese character, and the reasoning was entirely in simplified Chinese. I can't pin that on quantization without running the same prompt on unquantized Qwen3.8-27B, which I haven't done.

---

Also in this series:

• #14 Two modded 2080 Tis running Qwen3.8-27B with llama.cpp tensor parallel
• #12 Why isn't your 4-bit quant faster on a 2080 Ti?
• #20 FastLLM + DFlash2 on two 2080 Tis