# Serial loading reclaims 4.2 GB on 8 GB devices Two 7B Q4 models exceed an 8 GB ceiling when loaded concurrently. Serial residency via llama-swap reclaims 4.2 GB of KV cache headroom. Canonical page: https://makerportal.ai/blog/itria-github-867433720 Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-08-15 Section: Field note / Local LLMs · 6 min read Tags: itria, local-llm --- The 7B Q4 + 7B Q4 pair that [our prior measurement](/blog/thumb-dash-hackernews-49279500) found infeasible on an 8 GB ceiling—weights alone consume 8.4 GB—becomes viable under serial model loading. One model resident at a time leaves 3.8 GB for KV cache and embeddings, against the −0.4 GB the concurrent case produces. The recovered headroom is 4.2 GB, equal to W_idle exactly—the non-active model's weight size—which is why the swap-vs-concurrent decision is a hard binary threshold, not a continuous tradeoff. ## What llama-swap claims llama-swap describes itself as a Go reverse proxy managing process lifecycle for any OpenAI- or Anthropic-compatible inference server. The stated operational claims, with conditions: - **On-demand model switching** across llama.cpp forks, vllm, stable-diffusion.cpp, audio.cpp, and ComfyUI. No swap latency figure is given. - **TTL-based automatic unloading**: models are evicted after a configurable idle timeout. No accounting for mid-request eviction behaviour is specified. - **Concurrent models via a "swap matrix" DSL**: llama-swap states a custom DSL controls which models may run simultaneously, but publishes no VRAM guidance for choosing which pairs to allow. - **Anthropic API support** is limited to `v1/messages` and `v1/messages/count_tokens` only; no claim of full API parity is made. - **Zero binary dependencies**: this refers to the proxy binary itself. The inference server it wraps carries its own dependency surface. - **Unified nightly container** with llama-server, ik-llama-server, stable-diffusion.cpp, and whisper.cpp — scoped to CUDA and Vulkan targets only. - **Logs endpoint** buffers a fixed maximum of 10 KB. No throughput, latency, or memory overhead figures for the proxy layer itself are published anywhere in the project. ## Analysis llama-swap's core function is trading concurrent memory residency for serial residency. The memory available for KV cache and embeddings in each regime: $$ KV_\text{max}^\text{serial} = M - W_\text{active}, \qquad KV_\text{max}^\text{concurrent} = M - W_1 - W_2 $$ The swap dividend—headroom a serial proxy creates over concurrent loading—is therefore: $$ \Delta_{KV} = KV_\text{max}^\text{serial} - KV_\text{max}^\text{concurrent} = (M - W_\text{active}) - (M - W_1 - W_2) = W_\text{idle} $$ Substituting the 7B Q4 figures from [our measurement](/blog/thumb-dash-hackernews-49279500) (M = 8.0 GB, W = 4.2 GB for each model): $$ \Delta_{KV} = (8.0 - 4.2) - (8.0 - 4.2 - 4.2) = 3.8 - (-0.4) = 4.2 \text{ GB} $$ KV_max(concurrent) goes negative. That sign flip is what makes this a hard threshold rather than an optimisation knob—4.2 GB separates a system that cannot load at all from one with 3.8 GB of working headroom. | Model pair | W₁ (GB) | W₂ (GB) | W₁ + W₂ (GB) | KV budget, concurrent (GB) | KV budget, serial — larger model active (GB) | Swap mode | Conditions | |---|---|---|---|---|---|---|---| | 7B Q4 + 7B Q4 | 4.2 | 4.2 | 8.4 | −0.4 (infeasible) | 3.8 | **Required** | GGUF file sizes from studio measurement; embed excluded from residual | | 7B Q4 + 3B Q4 | 4.2 | 1.8 | 6.0 | 2.0 | 3.8 | Optional | Same | | 3B Q4 + 3B Q4 | 1.8 | 1.8 | 3.6 | 4.4 | 6.2 | Optional | Same | The "swap mode" classification follows directly from the sign of KV_max(concurrent). Where it goes negative, a serial proxy is a prerequisite. Only the 7B + 7B pair crosses that line at the 8 GB ceiling. For 7B + 3B, serialisation raises the KV budget from 2.0 GB to 3.8 GB—an improvement, but concurrent loading was already feasible. For 3B + 3B, the headroom gain is 1.8 GB against a concurrent budget that was already a comfortable 4.4 GB. The swap matrix DSL—which lets specific pairs run simultaneously—is safe only where W₁ + W₂ ≤ M. That condition holds for the 7B + 3B and 3B + 3B rows. It fails for 7B + 7B. llama-swap has no documented guard validating weight sizes against available memory when a swap matrix entry is parsed; the failure mode is an inference server that cannot load, not a proxy-layer rejection. The TTL eviction feature introduces an amortisation question the README skips: if a model unloads before the next request arrives, that request pays a full reload penalty. The 10 KB log buffer adds a second constraint—once a high-churn session saturates it, the swap event history is gone. The `/metrics` Prometheus endpoint is the only durable signal for swap frequency at scale; no substitute exists in the documented interface. The Anthropic claim warrants a close read. llama-swap lists `v1/messages` and `v1/messages/count_tokens` as the supported endpoints. Tool-use responses, batch endpoints, and streaming variants are unmentioned. Routing Anthropic-format client traffic through llama-swap without auditing the actual call surface first is a silent compatibility risk. ## What cannot be concluded Proxy-layer overhead—latency injected between client and inference server—is uncharacterised. The "zero dependencies / built for performance" claim in the README refers to the binary's dependency graph, not measured request latency. For a 7B Q4 model generating at single-digit tok/s, proxy overhead is almost certainly negligible. For a high-throughput vllm deployment, no measurement exists to bound it either way. ## What this means for itria Itria runs llama.cpp with Metal acceleration directly on-device. There is no server process to proxy, so llama-swap is not a production component here. The structural signal still applies: any two-model on-device workflow at or near the 8 GB unified-memory boundary—base iPhone 15 Pro, entry M1 iPad—must serialise model residency when both models are 7B Q4. A llama.cpp-internal equivalent—unloading one model context before initialising another—is already how itria would handle this case, and the derivation above quantifies what that serialisation buys: 4.2 GB of KV headroom the concurrent case cannot provide at all. For the 7B + 3B pairing [we already measured](/blog/thumb-dash-hackernews-49279500), serialisation raises the available KV budget from 2.0 GB to 3.8 GB—a 90% increase—which matters for long-context reasoning tasks where context window pressure is the primary quality constraint on a constrained device. --- *Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of [mostlygeek/llama-swap](https://github.com/mostlygeek/llama-swap) and this studio's own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (17 claim(s) and 0 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (2 derived from it, 15 row(s) supplied from our own tables). The studio has not re-run mostlygeek/llama-swap's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Does llama-swap flush GPU memory between swaps, or does it rely on the inference server to release it? llama-swap manages residency by starting and stopping the inference server process via `cmd` and `cmdStop`. It does not interact with the GPU memory allocator directly. Memory release depends entirely on the inference server's own shutdown behaviour; the README states process lifecycle management through these fields but gives no guarantee about residual VRAM state after `cmdStop` returns. ### Can the swap matrix DSL be configured to keep two 7B Q4 models concurrently resident on an 8 GB device? The configuration syntax permits it. The memory arithmetic does not: 4.2 + 4.2 = 8.4 GB exceeds the 8.0 GB ceiling before any KV cache is allocated. llama-swap documents no validation of model weight sizes against available memory when parsing a swap matrix entry. The failure lands at the inference server, not the proxy. ### What is the actual wall-clock cost of a swap event? llama-swap publishes no swap latency figures. The cost is dominated by the inference server's model-load time—a function of storage bandwidth, file size, and whether weights come from fast flash or a network mount. The proxy adds only a process-launch round-trip. On a modern system that's on the order of tens of milliseconds, small relative to the multi-gigabyte file load. ### Is the 10 KB log buffer configurable? The README gives no configuration knob—10 KB is a fixed limit. Engineers who need durable swap-event records should treat it as a hard ceiling and pair the `/metrics` Prometheus endpoint with an external time-series store.