llama.cpp fork streams KV cache on 16 GB CUDA

A llama.cpp fork adds block-granular KV cache streaming to the CUDA server path, and the one thing an engineer working from upstream would not assume is that the mechanism is not UVM: it is an explicit pinned-host-memory store with a bounded CUDA pool that splits between resident KV pages and a transfer ring, sized by a single flag.

What it is

The repository is llama.cpp-adaptive-kv-streaming, an experimental branch on top of the upstream llama.cpp CUDA path. It is C/C++, builds to build/bin/llama-server, and requires a CUDA GPU. The feature is gated behind --kv-stream-stage-mib N, where N is the pool size in MiB.

The mechanism, as the README describes it: authoritative KV tensors live in pinned host memory. A bounded CUDA pool of size N is shared by two consumers — resident KV pages (the pages the attention kernel reads directly from VRAM) and a transfer ring (staging space for pages being moved in or out). The runtime keeps as many pages resident as the budget allows, reclaims resident space for staging when more streaming is needed, and prefetches later layers while the current layer computes. The README states this “avoids relying on uncontrolled Unified Memory page thrashing and preserves exact attention over the full context.”

The branch also relaxes the upstream constraint that -b and -ub must both be 256. Any positive -ub no larger than -b is accepted. The Qwen3.8 MMA prefill path processes the full physical batch and allocates only the partial workspace the kernel emits; generic vector and F16-conversion fallback paths use a bounded 256-query workspace.

Fact	Value / condition	Source
Primary validation GPU	RTX 5070 Ti, 16 GB VRAM	README, “Adaptive KV Streaming”
Model	unsloth/Qwen3.8-27B-GGUF, UD-Q3_K_XL	README, “Adaptive KV Streaming”
Context length (primary)	262144 tokens	README, “Adaptive KV Streaming”
KV cache types	Q8_0 K, Q4_0 V	README, “Adaptive KV Streaming”
Attention	Flash Attention	README, “Adaptive KV Streaming”
Server slots	1	README, “Adaptive KV Streaming”
Pool size in example command	2304 MiB	README, example --kv-stream-stage-mib
Production-shaped run	122880 tokens at b/ub 512/512, streaming active	README, “Batch and micro-batch sizes”
b/ub values exercised (Q8_0/Q4_0 Qwen)	256/256, 512/512, 768/512, 1024/1024	README, “Batch and micro-batch sizes”
Benchmark sweep range	8K through requested max (example: 192K)	README, “Recreate the benchmark graph”
UVM requirement	Optional; does not change KV pool allocation	README, “Optional Unified Memory”
CUDA correctness tests	Cover every KV type accepted by CLI, incl. F16 fallback	README, “Adaptive KV Streaming”
Broader characterization	Not yet done for other models, KV combos, parallel slots, non-CUDA	README, “Adaptive KV Streaming”

Performance claims and their conditions

The README does not report tokens-per-second or latency figures. What it reports is completion: a 122880-token production-shaped run at 512/512 completed with adaptive streaming active, and the benchmark driver (which sweeps from 8K to a configured maximum, example 192K) exists to generate throughput graphs but the README does not inline the results.

The claim of “production-validated” is attached to the specific configuration: RTX 5070 Ti, 16 GB, Qwen3.8-27B UD-Q3_K_XL, 262144-token context, Flash Attention, Q8_0/Q4_0 KV, one slot. The README immediately follows with: “Production performance for other models, KV combinations, parallel slots, and non-CUDA backends is not yet broadly characterized.” So the validation scope is one GPU, one model, one quant, one KV type pair, one slot.

The correctness claim is stronger: CUDA correctness tests cover every KV type the CLI accepts, including F16-conversion fallback paths. That means the streaming path is not expected to change numerical results relative to a non-streaming run at the same context length — it is a memory-management change, not an approximation.

Reported by RaymondHuang210129/llama.cpp-ad… tokens· scroll →

Conditions around the comparison

scroll →

16 GB

video memory of the RTX 5070 Ti

2304 MiB

value of --kv-stream-sta

256 queries

bounded workspace size

192K

maximum context

What it is useful for

If you have a single CUDA GPU in the 12–16 GB range, a model whose weights fit but whose full KV cache at your target context does not, and you are running one slot, this is a concrete path to longer context than the naive “it fits or it doesn’t” check. The pinned-host-memory design means you are not at the mercy of the OS page-fault handler; the transfer ring gives you a bounded worst-case stall per layer.

It is not a replacement for CPU offload in the general sense. It is a KV-cache-specific streaming path. Model weights still need to fit in VRAM (or use UVM separately — the two are orthogonal). It does not help if your weights do not fit. It does not help on non-CUDA backends. It does not help with multi-slot serving in any characterized way.

For the reader who wants to sanity-check whether their own model/quant/context combination would leave enough VRAM for a pool after weights: the LLM GPU Memory Calculator will give you the KV cache size per token for a given KV type so you can see whether the gap between “weights fit” and “full KV fits” is large enough to warrant the streaming overhead.

It is also worth noting alongside buun-llama-cpp’s VBR approach, which attacks the same “long context on limited VRAM” problem from the inference-step side rather than the memory side. The two are not in competition; they address different bottlenecks.

What cannot be concluded

The README presents one completed run and a benchmark driver, but no throughput numbers, no latency percentiles, no comparison against a UVM baseline at the same context length, and no measurement of the transfer-ring stall as context grows within the 262K window. The claim “preserves exact attention” is a correctness claim (the kernel reads the same values), not a performance claim. Whether the streaming overhead makes 262K context practical at interactive latency on a 5070 Ti is not stated.

---

Method: this note was drafted by qwen/qwen3.8-27b from a single source — the published README of RaymondHuang210129/llama.cpp-adaptive-kv-streaming. Before publication an automated gate re-checked every extracted claim against the source document (29 claim(s) and 7 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (1 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run RaymondHuang210129/llama.cpp-adaptive-kv-streaming’s benchmarks; figures attributed to it are its own.

FAQ

Does this work on Apple silicon or AMD GPUs?

No. The branch modifies the CUDA llama-server path. The README makes no mention of Metal, ROCm, or Vulkan support for the streaming feature. The upstream llama.cpp supports those backends, but the adaptive KV streaming code is CUDA-specific.

Can I use this with a model that doesn't fit in VRAM at all?

Not directly. The README states the feature is "intended for running long contexts when model weights leave too little VRAM for the complete KV cache." The weights must still be resident (or use UVM separately). If the weights themselves do not fit, you need a different solution — smaller quant, CPU offload, or a smaller model.

What does the 2304 MiB pool size mean in practice?

It is the total CUDA memory allocated for the streaming pool: resident KV pages plus the transfer ring, sharing that budget. The runtime adapts the split as context grows. The README does not state what fraction ends up as resident pages versus ring at 262K context, nor does it state what happens if the pool is too small for the configured context — presumably the run fails or the context is capped, but the README does not specify.

Is this safe to use in a production serving environment?

The README calls it "research code" and explicitly states that production performance is not broadly characterized beyond the single validated configuration. It is one slot, one model, one GPU. If your production environment matches that configuration exactly and you have validated the latency characteristics yourself, the correctness tests give you confidence that outputs are numerically identical to a non-streaming run. For anything else, treat it as experimental.