buun-llama-cpp VBR uses 10 steps per layer

The step-count formula in buun-llama-cpp’s two published examples — 160 steps for a 16-layer model, 600 for a 60-layer model — reduces to exactly 10 degradation steps per attention layer. That’s the product of 6 codec tiers applied independently to two KV sides. The README doesn’t state it; dividing both figures by their respective layer counts is the only way to recover it.

What buun-llama-cpp reports

The fork introduces VBR KV cache management, degrading cache quality one (layer, side) tensor at a time under VRAM pressure. Six tiers form the codec ladder: f16 → turbo8 → turbo4 → turbo3_tcq → turbo2_tcq → turbo1_tcq. buun-llama-cpp states this produces 160 distinct quality steps for Qwen35 (16 attention layers) and 600 for Gemma4-31B (60 layers), with a default floor at turbo4.

For TCQ, the fork reports: Viterbi encoding on a 512-state trellis cuts median KL-divergence by ~40% versus scalar quantization at the 3-bit setting, using 3.25 bpv against scalar’s 3.50 bpv, with perplexity on par with an f16 KV cache. Decode is O(1) via a sliding-window bit lookup. Viterbi cost is paid at cache-fill time only. On weaker-compute hardware, buun-llama-cpp reports TCQ types can be up to ~40% slower than scalar counterparts at decode. VBR requires a CUDA or ROCm backend; CPU-resident KV layers fall back to q8_0, and flash attention is force-enabled.

Analysis

The step-count formula

Let L be the number of attention layers and T the number of tiers in the codec ladder. Each layer has two sides (K and V), each capable of T − 1 downward transitions:

$$\text{steps}(L) = L \times 2 \times (T - 1)$$

With T = 6 and therefore T − 1 = 5, substituting the supplied figures:

$$\text{Qwen35:}\quad 16 \times 2 \times 5 = 160$$ $$\text{Gemma4-31B:}\quad 60 \times 2 \times 5 = 600$$

Both reproduce exactly. The factor of 10 per layer is not approximate. VRAM reclaimed per step equals one layer-side’s worth of cache at the tier boundary being crossed, spread across 2 × (T − 1) = 10 steps. Shallower models produce coarser steps per reclaimed byte; deeper models track the budget ceiling with proportionally more precision. In a session that gradually fills VRAM, a 16-layer model has 10 adjustment opportunities per layer and 160 total; a 60-layer model has the same 10 per layer but 600 total. The resolution difference is purely a function of L, not architecture.

Codec ladder: implied bits per value

buun-llama-cpp’s compression claims, with the f16 baseline of 16 bpv, imply an effective bpv per tier via 16 ÷ compression_ratio. The turbo3_tcq derived value (3.20) and the separately stated TCQ bpv (3.25) differ by 0.05 — within rounding of the ~5× compression figure, and consistent with minor trellis overhead.

Tier	Compression (claimed)	Implied bpv (16 ÷ ratio)	KLD vs scalar	Conditions on claim
f16	1× reference	16.00	—	Baseline
turbo8	Not quantified	—	—	Codec ladder position only
turbo4	~3.8×	4.21	Not reported	No speed penalty claimed; CUDA/ROCm required
turbo3_tcq	~5×	3.20	~40% lower (at stated 3.25 bpv)	512-state trellis; Viterbi at encode time
turbo2_tcq	~7×	2.29	Not reported at symmetric depth	Asymmetric 2.75 bpv variant 15–17% lower KLD than reverse
turbo1_tcq	Not quantified	—	Not reported	Lowest rung; conditions unstated

The asymmetric assignment result is the most actionable entry in this table. buun-llama-cpp reports 3-bit keys plus 2-bit values (2.75 bpv average) yields 15–17% lower KLD than the reverse, attributed to adaptive alpha compensating V quantization error. K tensors are more bit-sensitive than V tensors; the asymmetric allocation exploits this without changing the average bit count.

DSpark GPU assist efficiency

buun-llama-cpp reports a dual-3090 DSpark result improving from 48.7 to 51.5 tok/s after placing lightweight draft layers and the Markov/output tail in ~594 MiB of GPU memory. The absolute gain is 51.5 − 48.7 = 2.8 tok/s, a 5.75% improvement. Total process RSS is 95.3 GiB; 594 MiB represents 594 ÷ (95.3 × 1024) ≈ 0.61% of total footprint. Getting a 5.75% throughput gain from 0.61% of footprint on GPU points to access latency as the bottleneck for those draft components, not data volume — consistent with dispatch overhead for infrequent but latency-critical operations on a CPU path.

What cannot be concluded

buun-llama-cpp publishes KLD and perplexity data only for the turbo3_tcq 3-bit setting. The ~7× and ~3.8× compression figures for turbo2_tcq and turbo4 carry no attached perplexity or KLD measurements. Whether either meets quality requirements for a production workload is unresolvable from the published claims. The ROCm results are explicitly conditioned on ROCm 7.2 with gfx1100 (RX 7900 XTX); behavior on other RDNA generations or ROCm versions is not stated.

What this means for Thumbdash

Thumbdash runs SmolLM2-360M-Instruct via llama.cpp on Metal. buun-llama-cpp requires CUDA or ROCm for VBR and TCQ; Metal builds fall back to q8_0 for any CPU-resident KV layers. VBR and TCQ are not available in the current Metal-backed pipeline.

SmolLM2-360M is shallow. Applying the step-count formula to its layer count would yield fewer than 160 VBR steps, giving coarser quality-per-reclaimed-byte resolution than either of buun-llama-cpp’s example models. For Thumbdash’s game loop — short, high-frequency sessions with no long-session accumulation — that coarseness isn’t a penalty: the KV cache never grows large enough to trigger degradation pressure in the first place. The static weight budget addressed in our serial loading note remains the binding constraint at 8 GB. The --mmproj-gpu-swap result (22.6 GiB of Qwen3.6-27B Q6_K + MTP leaving no headroom for mmproj’s 1.1 GiB on a 24 GiB RTX 3090) is not directly applicable but frames the memory arithmetic for any future multimodal expansion of the on-device pipeline.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of spiritbuun/buun-llama-cpp and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (22 claim(s) and 0 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (10 derived from it, 24 row(s) supplied from our own tables). The studio has not re-run spiritbuun/buun-llama-cpp’s benchmarks; figures attributed to it are its own.

FAQ

Does TCQ's O(1) decode claim mean no runtime penalty versus scalar quantization?

No. buun-llama-cpp states the O(1) sliding-window decode replaces trellis traversal at inference, but separately reports TCQ types can be up to ~40% slower than scalar counterparts on weaker-compute hardware. The O(1) claim is about asymptotic complexity, not wall-clock parity; the constant factor is larger than scalar, and that constant is exposed when the hardware cannot cover it with parallelism.

Can VBR be used on a Metal or CPU-only build of llama.cpp?

Not in dynamic mode. buun-llama-cpp explicitly requires a CUDA or ROCm backend for the TurboQuant interface; layers whose KV resides on CPU fall back to q8_0. Flash attention is force-enabled by VBR. A Metal-only build forgoes all turbo tiers and retains only the q8_0 fallback regardless of VRAM pressure.

What happens to server session save-restore under VBR?

buun-llama-cpp disables context-shift, self-extend, and slot/session save-restore in dynamic VBR mode. The stated reason is that snapshotting a tier-typed KV cache cannot be reliably restored across a degradation boundary — the tier assignment at snapshot time may not match available VRAM on restore. Deployments that depend on session persistence need to run a fixed-tier mode or accept that sessions cannot survive restarts.

Is the 160-step / 600-step count for the full session lifetime or per context window?

Per session lifetime. buun-llama-cpp describes VBR as responding to budget pressure as the session grows, moving one (layer, side) tensor down one tier at a time with no periodic reset. The step count is the total number of quality states available across the entire accumulated KV cache, not a per-window or rolling measure. Once a tier boundary is crossed downward, that layer-side does not automatically recover when tokens are evicted.