# llama.cpp fork streams KV cache on 16 GB CUDA A llama.cpp fork adds block-granular KV cache streaming to the CUDA server path using pinned host memory and a bounded pool, enabling 262K context on a… Canonical page: https://makerportal.ai/blog/itria-hackernews-49511882 Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-09-01 Section: Readout / Local LLMs · 6 min read Tags: local-llm --- A llama.cpp fork adds block-granular KV cache streaming to the CUDA server path, and the one thing an engineer working from upstream would not assume is that the mechanism is not UVM: it is an explicit pinned-host-memory store with a bounded CUDA pool that splits between resident KV pages and a transfer ring, sized by a single flag. ## What it is The repository is [llama.cpp-adaptive-kv-streaming](https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming), an experimental branch on top of the upstream llama.cpp CUDA path. It is C/C++, builds to `build/bin/llama-server`, and requires a CUDA GPU. The feature is gated behind `--kv-stream-stage-mib N`, where N is the pool size in MiB. The mechanism, as the README describes it: authoritative KV tensors live in pinned host memory. A bounded CUDA pool of size N is shared by two consumers — resident KV pages (the pages the attention kernel reads directly from VRAM) and a transfer ring (staging space for pages being moved in or out). The runtime keeps as many pages resident as the budget allows, reclaims resident space for staging when more streaming is needed, and prefetches later layers while the current layer computes. The README states this "avoids relying on uncontrolled Unified Memory page thrashing and preserves exact attention over the full context." The branch also relaxes the upstream constraint that `-b` and `-ub` must both be 256. Any positive `-ub` no larger than `-b` is accepted. The Qwen3.8 MMA prefill path processes the full physical batch and allocates only the partial workspace the kernel emits; generic vector and F16-conversion fallback paths use a bounded 256-query workspace. | Fact | Value / condition | Source | |---|---|---| | Primary validation GPU | RTX 5070 Ti, 16 GB VRAM | README, "Adaptive KV Streaming" | | Model | `unsloth/Qwen3.8-27B-GGUF`, UD-Q3_K_XL | README, "Adaptive KV Streaming" | | Context length (primary) | 262144 tokens | README, "Adaptive KV Streaming" | | KV cache types | Q8_0 K, Q4_0 V | README, "Adaptive KV Streaming" | | Attention | Flash Attention | README, "Adaptive KV Streaming" | | Server slots | 1 | README, "Adaptive KV Streaming" | | Pool size in example command | 2304 MiB | README, example `--kv-stream-stage-mib` | | Production-shaped run | 122880 tokens at b/ub 512/512, streaming active | README, "Batch and micro-batch sizes" | | b/ub values exercised (Q8_0/Q4_0 Qwen) | 256/256, 512/512, 768/512, 1024/1024 | README, "Batch and micro-batch sizes" | | Benchmark sweep range | 8K through requested max (example: 192K) | README, "Recreate the benchmark graph" | | UVM requirement | Optional; does not change KV pool allocation | README, "Optional Unified Memory" | | CUDA correctness tests | Cover every KV type accepted by CLI, incl. F16 fallback | README, "Adaptive KV Streaming" | | Broader characterization | Not yet done for other models, KV combos, parallel slots, non-CUDA | README, "Adaptive KV Streaming" | ## Performance claims and their conditions The README does not report tokens-per-second or latency figures. What it reports is completion: a 122880-token production-shaped run at 512/512 completed with adaptive streaming active, and the benchmark driver (which sweeps from 8K to a configured maximum, example 192K) exists to generate throughput graphs but the README does not inline the results. The claim of "production-validated" is attached to the specific configuration: RTX 5070 Ti, 16 GB, Qwen3.8-27B UD-Q3_K_XL, 262144-token context, Flash Attention, Q8_0/Q4_0 KV, one slot. The README immediately follows with: "Production performance for other models, KV combinations, parallel slots, and non-CUDA backends is not yet broadly characterized." So the validation scope is one GPU, one model, one quant, one KV type pair, one slot. The correctness claim is stronger: CUDA correctness tests cover every KV type the CLI accepts, including F16-conversion fallback paths. That means the streaming path is not expected to change numerical results relative to a non-streaming run at the same context length — it is a memory-management change, not an approximation.
Reported by RaymondHuang210129/llama.cpp-ad… tokens· scroll →
context length used 262,144 tokens token count 122,880 tokens

Conditions around the comparison

scroll →
16 GB
video memory of the RTX 5070 Ti
2304 MiB
value of --kv-stream-sta
256 queries
bounded workspace size
192K
maximum context
Figure: values as RaymondHuang210129/llama.cpp-adaptive-kv-streaming reports them, drawn from quantities extracted and quote-verified for this post. Bars are proportional within their shared unit; condition cells are exact values and are not scaled against one another. The studio has not re-measured them.
## What it is useful for If you have a single CUDA GPU in the 12–16 GB range, a model whose weights fit but whose full KV cache at your target context does not, and you are running one slot, this is a concrete path to longer context than the naive "it fits or it doesn't" check. The pinned-host-memory design means you are not at the mercy of the OS page-fault handler; the transfer ring gives you a bounded worst-case stall per layer. It is not a replacement for CPU offload in the general sense. It is a KV-cache-specific streaming path. Model weights still need to fit in VRAM (or use UVM separately — the two are orthogonal). It does not help if your weights do not fit. It does not help on non-CUDA backends. It does not help with multi-slot serving in any characterized way. For the reader who wants to sanity-check whether their own model/quant/context combination would leave enough VRAM for a pool after weights: the [LLM GPU Memory Calculator](/lab/llm-vram-kvcache-calculator) will give you the KV cache size per token for a given KV type so you can see whether the gap between "weights fit" and "full KV fits" is large enough to warrant the streaming overhead. It is also worth noting alongside [buun-llama-cpp's VBR approach](/blog/thumb-dash-github-1192318297), which attacks the same "long context on limited VRAM" problem from the inference-step side rather than the memory side. The two are not in competition; they address different bottlenecks. ## What cannot be concluded The README presents one completed run and a benchmark driver, but no throughput numbers, no latency percentiles, no comparison against a UVM baseline at the same context length, and no measurement of the transfer-ring stall as context grows within the 262K window. The claim "preserves exact attention" is a correctness claim (the kernel reads the same values), not a performance claim. Whether the streaming overhead makes 262K context practical at interactive latency on a 5070 Ti is not stated. --- *Method: this note was drafted by qwen/qwen3.8-27b from a single source — the published README of [RaymondHuang210129/llama.cpp-adaptive-kv-streaming](https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming). Before publication an automated gate re-checked every extracted claim against the source document (29 claim(s) and 7 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (1 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run RaymondHuang210129/llama.cpp-adaptive-kv-streaming's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Does this work on Apple silicon or AMD GPUs? No. The branch modifies the CUDA `llama-server` path. The README makes no mention of Metal, ROCm, or Vulkan support for the streaming feature. The upstream llama.cpp supports those backends, but the adaptive KV streaming code is CUDA-specific. ### Can I use this with a model that doesn't fit in VRAM at all? Not directly. The README states the feature is "intended for running long contexts when model weights leave too little VRAM for the complete KV cache." The weights must still be resident (or use UVM separately). If the weights themselves do not fit, you need a different solution — smaller quant, CPU offload, or a smaller model. ### What does the 2304 MiB pool size mean in practice? It is the total CUDA memory allocated for the streaming pool: resident KV pages plus the transfer ring, sharing that budget. The runtime adapts the split as context grows. The README does not state what fraction ends up as resident pages versus ring at 262K context, nor does it state what happens if the pool is too small for the configured context — presumably the run fails or the context is capped, but the README does not specify. ### Is this safe to use in a production serving environment? The README calls it "research code" and explicitly states that production performance is not broadly characterized beyond the single validated configuration. It is one slot, one model, one GPU. If your production environment matches that configuration exactly and you have validated the latency characteristics yourself, the correctness tests give you confidence that outputs are numerically identical to a non-streaming run. For anything else, treat it as experimental.