Serial loading reclaims 4.2 GB on 8 GB devices

The 7B Q4 + 7B Q4 pair that our prior measurement found infeasible on an 8 GB ceiling—weights alone consume 8.4 GB—becomes viable under serial model loading. One model resident at a time leaves 3.8 GB for KV cache and embeddings, against the −0.4 GB the concurrent case produces. The recovered headroom is 4.2 GB, equal to W_idle exactly—the non-active model’s weight size—which is why the swap-vs-concurrent decision is a hard binary threshold, not a continuous tradeoff.

What llama-swap claims

llama-swap describes itself as a Go reverse proxy managing process lifecycle for any OpenAI- or Anthropic-compatible inference server. The stated operational claims, with conditions:

• On-demand model switching across llama.cpp forks, vllm, stable-diffusion.cpp, audio.cpp, and ComfyUI. No swap latency figure is given.
• TTL-based automatic unloading: models are evicted after a configurable idle timeout. No accounting for mid-request eviction behaviour is specified.
• Concurrent models via a “swap matrix” DSL: llama-swap states a custom DSL controls which models may run simultaneously, but publishes no VRAM guidance for choosing which pairs to allow.
• Anthropic API support is limited to v1/messages and v1/messages/count_tokens only; no claim of full API parity is made.
• Zero binary dependencies: this refers to the proxy binary itself. The inference server it wraps carries its own dependency surface.
• Unified nightly container with llama-server, ik-llama-server, stable-diffusion.cpp, and whisper.cpp — scoped to CUDA and Vulkan targets only.
• Logs endpoint buffers a fixed maximum of 10 KB.

No throughput, latency, or memory overhead figures for the proxy layer itself are published anywhere in the project.

Analysis

llama-swap’s core function is trading concurrent memory residency for serial residency. The memory available for KV cache and embeddings in each regime:

$$KV_\text{max}^\text{serial} = M - W_\text{active}, \qquad KV_\text{max}^\text{concurrent} = M - W_1 - W_2$$

The swap dividend—headroom a serial proxy creates over concurrent loading—is therefore:

$$\Delta_{KV} = KV_\text{max}^\text{serial} - KV_\text{max}^\text{concurrent} = (M - W_\text{active}) - (M - W_1 - W_2) = W_\text{idle}$$

Substituting the 7B Q4 figures from our measurement (M = 8.0 GB, W = 4.2 GB for each model):

$$\Delta_{KV} = (8.0 - 4.2) - (8.0 - 4.2 - 4.2) = 3.8 - (-0.4) = 4.2 \text{ GB}$$

KV_max(concurrent) goes negative. That sign flip is what makes this a hard threshold rather than an optimisation knob—4.2 GB separates a system that cannot load at all from one with 3.8 GB of working headroom.

Model pair	W₁ (GB)	W₂ (GB)	W₁ + W₂ (GB)	KV budget, concurrent (GB)	KV budget, serial — larger model active (GB)	Swap mode	Conditions
7B Q4 + 7B Q4	4.2	4.2	8.4	−0.4 (infeasible)	3.8	Required	GGUF file sizes from studio measurement; embed excluded from residual
7B Q4 + 3B Q4	4.2	1.8	6.0	2.0	3.8	Optional	Same
3B Q4 + 3B Q4	1.8	1.8	3.6	4.4	6.2	Optional	Same

The “swap mode” classification follows directly from the sign of KV_max(concurrent). Where it goes negative, a serial proxy is a prerequisite. Only the 7B + 7B pair crosses that line at the 8 GB ceiling. For 7B + 3B, serialisation raises the KV budget from 2.0 GB to 3.8 GB—an improvement, but concurrent loading was already feasible. For 3B + 3B, the headroom gain is 1.8 GB against a concurrent budget that was already a comfortable 4.4 GB.

The swap matrix DSL—which lets specific pairs run simultaneously—is safe only where W₁ + W₂ ≤ M. That condition holds for the 7B + 3B and 3B + 3B rows. It fails for 7B + 7B. llama-swap has no documented guard validating weight sizes against available memory when a swap matrix entry is parsed; the failure mode is an inference server that cannot load, not a proxy-layer rejection.

The TTL eviction feature introduces an amortisation question the README skips: if a model unloads before the next request arrives, that request pays a full reload penalty. The 10 KB log buffer adds a second constraint—once a high-churn session saturates it, the swap event history is gone. The /metrics Prometheus endpoint is the only durable signal for swap frequency at scale; no substitute exists in the documented interface.

The Anthropic claim warrants a close read. llama-swap lists v1/messages and v1/messages/count_tokens as the supported endpoints. Tool-use responses, batch endpoints, and streaming variants are unmentioned. Routing Anthropic-format client traffic through llama-swap without auditing the actual call surface first is a silent compatibility risk.

What cannot be concluded

Proxy-layer overhead—latency injected between client and inference server—is uncharacterised. The “zero dependencies / built for performance” claim in the README refers to the binary’s dependency graph, not measured request latency. For a 7B Q4 model generating at single-digit tok/s, proxy overhead is almost certainly negligible. For a high-throughput vllm deployment, no measurement exists to bound it either way.

What this means for itria

Itria runs llama.cpp with Metal acceleration directly on-device. There is no server process to proxy, so llama-swap is not a production component here. The structural signal still applies: any two-model on-device workflow at or near the 8 GB unified-memory boundary—base iPhone 15 Pro, entry M1 iPad—must serialise model residency when both models are 7B Q4. A llama.cpp-internal equivalent—unloading one model context before initialising another—is already how itria would handle this case, and the derivation above quantifies what that serialisation buys: 4.2 GB of KV headroom the concurrent case cannot provide at all. For the 7B + 3B pairing we already measured, serialisation raises the available KV budget from 2.0 GB to 3.8 GB—a 90% increase—which matters for long-context reasoning tasks where context window pressure is the primary quality constraint on a constrained device.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of mostlygeek/llama-swap and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (17 claim(s) and 0 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (2 derived from it, 15 row(s) supplied from our own tables). The studio has not re-run mostlygeek/llama-swap’s benchmarks; figures attributed to it are its own.

FAQ

Does llama-swap flush GPU memory between swaps, or does it rely on the inference server to release it?

llama-swap manages residency by starting and stopping the inference server process via cmd and cmdStop. It does not interact with the GPU memory allocator directly. Memory release depends entirely on the inference server's own shutdown behaviour; the README states process lifecycle management through these fields but gives no guarantee about residual VRAM state after cmdStop returns.

Can the swap matrix DSL be configured to keep two 7B Q4 models concurrently resident on an 8 GB device?

The configuration syntax permits it. The memory arithmetic does not: 4.2 + 4.2 = 8.4 GB exceeds the 8.0 GB ceiling before any KV cache is allocated. llama-swap documents no validation of model weight sizes against available memory when parsing a swap matrix entry. The failure lands at the inference server, not the proxy.

What is the actual wall-clock cost of a swap event?

llama-swap publishes no swap latency figures. The cost is dominated by the inference server's model-load time—a function of storage bandwidth, file size, and whether weights come from fast flash or a network mount. The proxy adds only a process-launch round-trip. On a modern system that's on the order of tens of milliseconds, small relative to the multi-gigabyte file load.

Is the 10 KB log buffer configurable?

The README gives no configuration knob—10 KB is a fixed limit. Engineers who need durable swap-event records should treat it as a hard ceiling and pair the /metrics Prometheus endpoint with an external time-series store.