Two 7B Q4 models exceed DLLM's 8 GB VRAM

Two measured 7B Q4 GGUF files occupy 8.4 GB combined — 0.4 GB above DLLM’s stated 8 GB VRAM ceiling. That rules out any two-7B-model configuration and puts the embed model and KV-cache budget in direct competition for whatever the agent and summary models leave behind.

What DLLM claims

DLLM reports a three-model pipeline (agent, summary, embed) requiring 8 GB VRAM for its described model setup, with full CUDA offloading. The project builds against llama.cpp via importC — D’s mechanism for consuming C headers directly at compile time — and claims this eliminates Python, bindings, and overhead. Multimodal input runs via mtmd; context management relies on KV-cache condensation; retrieval is handled by the embed model. Tools are auto-registered through @Tool("Description") user-defined attributes on D functions; included tools span file I/O, Docker-sandboxed execution, web search, and audio playback. DLLM states it compiles with DMD, LDC, or GDC.

Analysis

The 8 GB ceiling binds directly on model weight allocation. With all three models resident in VRAM simultaneously, the capacity constraint is:

$$W_{agent} + W_{summary} + W_{embed} + KV_{budget} \leq VRAM_{total}$$

Our published hardware measurements give Q4 GGUF sizes of 1.8 GB (3B-class) and 4.2 GB (7B-class). Substituting the largest plausible agent-and-summary pairing — both 7B Q4:

$$4.2 + 4.2 + W_{embed} + KV_{budget} \leq 8.0 \implies W_{embed} + KV_{budget} \leq -0.4 \text{ GB}$$

Infeasible before counting either the embed model or a single KV-cache entry. Substituting agent = 7B Q4, summary = 3B Q4:

$$4.2 + 1.8 + W_{embed} + KV_{budget} \leq 8.0 \implies W_{embed} + KV_{budget} \leq 2.0 \text{ GB}$$

A 3B Q4 embed model (1.8 GB) leaves only 0.2 GB for KV-cache — effectively no usable context. The 7B + 3B pairing works only if the embed model is purpose-built and well under 1 GB.

Agent	Summary	W_agent (GB)	W_summary (GB)	Two-model total (GB)	Residual for embed + KV (GB)	Within 8 GB ceiling	Conditions
7B Q4	7B Q4	4.2	4.2	8.4	−0.4	No	GGUF file sizes from our measurement; embed and KV excluded
7B Q4	3B Q4	4.2	1.8	6.0	2.0	Yes	GGUF file sizes from our measurement; embed and KV excluded
3B Q4	3B Q4	1.8	1.8	3.6	4.4	Yes	GGUF file sizes from our measurement; embed and KV excluded

With agent = 3B Q4 and summary = 3B Q4, the 4.4 GB residual accommodates a compact embed model plus meaningful KV-cache. That combination decodes at 37 tok/s on a Jetson Orin Nano (68 GB/s unified memory) or 560 tok/s on an RTX 4090 (1,008 GB/s GDDR6X), per our measurements. Swapping to a 7B Q4 agent on the RTX 4090 drops decode to 240 tok/s — still within the 8 GB ceiling given a 3B summary and a compact embed.

The importC overhead claim, disaggregated. importC is a compile-time facility: D’s compiler parses llama.cpp’s C headers directly and emits ordinary C ABI calls — no interpreter, no ctypes layer, no Python runtime in the process image. The per-token call overhead it eliminates is real. It is also measured in nanoseconds. As our bandwidth-utilization measurements show, llama.cpp running natively on an RTX 4090 or M4 Max already saturates rated VRAM bandwidth at 99–100%; there is no headroom a faster call path could recover. The meaningful advantages are deployment-graph simplicity (no Python environment, no Conda, no binding layer to version-pin), cold-start latency, and binary size — none of which register in a tok/s figure.

The @Tool("Description") UDA approach has a structural implication worth noting. In D, user-defined attributes are resolved at compile time, meaning tool discovery happens during compilation rather than at agent startup. This is architecturally link-time registration, not runtime reflection. For the Docker-sandboxed execution tool specifically, the sandbox entry points are statically known and auditable — relevant for reproducible builds, though not a substitute for sandbox escape analysis.

What cannot be concluded

DLLM does not publish which specific models populate its three-model setup. The 8 GB figure fits several combinations — a 7B agent with a compact summary and a sub-1 GB embed model, or three smaller models with a generous KV-cache allocation. The table above bounds the feasible space from our measured GGUF sizes; it does not identify which combination DLLM’s author actually uses. The importC assessment covers only call-dispatch and runtime overhead — prefill latency, CUDA stream contention across three concurrent allocations, and the throughput cost of KV-cache condensation passes are outside its scope.

Thumbdash

Thumbdash runs SmolLM2-360M-Instruct through llama.cpp on Metal — a single model whose entire weight fits inside the KV-cache residual from even the most constrained pipeline combination in the table above. The two projects target different problems. DLLM optimises for context-window capacity and multi-model coordination on a CUDA device; Thumbdash optimises for sub-100 ms keystroke-response latency on-device, with no VRAM contention between models. DLLM’s @Tool compile-time registration and KV-cache condensation have no direct application in Thumbdash’s single-turn loop, but where multi-model pipeline complexity earns its cost is covered in our notes on agentic DSP verification.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of DannyArends/DLLM and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (10 claim(s) and 1 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (9 derived from it, 10 row(s) supplied from our own tables). The studio has not re-run DannyArends/DLLM’s benchmarks; figures attributed to it are its own.

FAQ

Does importC actually make DLLM faster than a Python-binding equivalent?

Not in tokens per second. On a CUDA device, inference throughput is bounded by VRAM bandwidth and model weight volume — importC changes neither. The advantage is deployment-graph simplicity: no Python interpreter, no binding layer to version-pin, no serialisation overhead between the D process and llama.cpp. Cold-start time and binary footprint are different; those improve. Sustained decode rate does not.

Are all three models resident in VRAM simultaneously?

DLLM claims 8 GB VRAM for the full described setup alongside CUDA offloading. If models were swapped in and out per pipeline stage, the 8 GB figure would instead describe peak single-model VRAM — a much weaker and nearly uninformative constraint. The natural reading of "the above model setup requires 8 GB VRAM" is that all three weight sets are resident concurrently. That is the assumption the weight-budget analysis uses.

What decode rate should I expect for the agent model on an 8 GB VRAM card?

Decode rate scales with VRAM bandwidth, and llama.cpp consistently saturates it. An 8 GB consumer card is not the RTX 4090 at 1,008 GB/s, so the 240 tok/s figure for 7B Q4 on that device does not apply directly. What does hold: KV-cache condensation passes — which re-run part of the model to compress context — consume bandwidth from the same budget as decode. Condensation frequency is therefore a direct lever on effective throughput on bandwidth-limited hardware.

Does KV-cache condensation change the weight-budget analysis?

No. KV-cache condensation reduces the KV-cache footprint at runtime by evicting or summarising stored key-value pairs; it does not touch the loaded model weights W_i for any of the three models. Condensation shrinks the KV_budget term in the inequality — useful for sustaining longer conversations within a fixed VRAM ceiling — but the weight allocation is unchanged.