# Two 7B Q4 models exceed DLLM's 8 GB VRAM

Measured Q4 GGUF sizes show two 7B models consume 8.4 GB, 0.4 GB over DLLM's ceiling, leaving no room for the embed model or KV-cache.

Canonical page: https://makerportal.ai/blog/thumb-dash-hackernews-49279500
Author: Joshua Hrisko, Principal Engineer — MakerPortal
Published: 2026-08-13
Section: Field note / Local LLMs · 5 min read
Tags: thumbdash, local-llm

---

Two measured 7B Q4 GGUF files occupy 8.4 GB combined — 0.4 GB above DLLM's stated 8 GB VRAM ceiling. That rules out any two-7B-model configuration and puts the embed model and KV-cache budget in direct competition for whatever the agent and summary models leave behind.

## What DLLM claims

DLLM reports a three-model pipeline (agent, summary, embed) requiring 8 GB VRAM for its described model setup, with full CUDA offloading. The project builds against llama.cpp via importC — D's mechanism for consuming C headers directly at compile time — and claims this eliminates Python, bindings, and overhead. Multimodal input runs via mtmd; context management relies on KV-cache condensation; retrieval is handled by the embed model. Tools are auto-registered through `@Tool("Description")` user-defined attributes on D functions; included tools span file I/O, Docker-sandboxed execution, web search, and audio playback. DLLM states it compiles with DMD, LDC, or GDC.

## Analysis

The 8 GB ceiling binds directly on model weight allocation. With all three models resident in VRAM simultaneously, the capacity constraint is:

$$
W_{agent} + W_{summary} + W_{embed} + KV_{budget} \leq VRAM_{total}
$$

Our [published hardware measurements](/blog/2026-08-11-itria-hackernews-49245161) give Q4 GGUF sizes of 1.8 GB (3B-class) and 4.2 GB (7B-class). Substituting the largest plausible agent-and-summary pairing — both 7B Q4:

$$
4.2 + 4.2 + W_{embed} + KV_{budget} \leq 8.0 \implies W_{embed} + KV_{budget} \leq -0.4 \text{ GB}
$$

Infeasible before counting either the embed model or a single KV-cache entry. Substituting agent = 7B Q4, summary = 3B Q4:

$$
4.2 + 1.8 + W_{embed} + KV_{budget} \leq 8.0 \implies W_{embed} + KV_{budget} \leq 2.0 \text{ GB}
$$

A 3B Q4 embed model (1.8 GB) leaves only 0.2 GB for KV-cache — effectively no usable context. The 7B + 3B pairing works only if the embed model is purpose-built and well under 1 GB.

| Agent | Summary | W\_agent (GB) | W\_summary (GB) | Two-model total (GB) | Residual for embed + KV (GB) | Within 8 GB ceiling | Conditions |
|---|---|---|---|---|---|---|---|
| 7B Q4 | 7B Q4 | 4.2 | 4.2 | 8.4 | −0.4 | No | GGUF file sizes from our measurement; embed and KV excluded |
| 7B Q4 | 3B Q4 | 4.2 | 1.8 | 6.0 | 2.0 | Yes | GGUF file sizes from our measurement; embed and KV excluded |
| 3B Q4 | 3B Q4 | 1.8 | 1.8 | 3.6 | 4.4 | Yes | GGUF file sizes from our measurement; embed and KV excluded |

With agent = 3B Q4 and summary = 3B Q4, the 4.4 GB residual accommodates a compact embed model plus meaningful KV-cache. That combination decodes at 37 tok/s on a Jetson Orin Nano (68 GB/s unified memory) or 560 tok/s on an RTX 4090 (1,008 GB/s GDDR6X), per our measurements. Swapping to a 7B Q4 agent on the RTX 4090 drops decode to 240 tok/s — still within the 8 GB ceiling given a 3B summary and a compact embed.

**The importC overhead claim, disaggregated.** importC is a compile-time facility: D's compiler parses llama.cpp's C headers directly and emits ordinary C ABI calls — no interpreter, no ctypes layer, no Python runtime in the process image. The per-token call overhead it eliminates is real. It is also measured in nanoseconds. As our [bandwidth-utilization measurements](/blog/biquadia-hackernews-49259339) show, llama.cpp running natively on an RTX 4090 or M4 Max already saturates rated VRAM bandwidth at 99–100%; there is no headroom a faster call path could recover. The meaningful advantages are deployment-graph simplicity (no Python environment, no Conda, no binding layer to version-pin), cold-start latency, and binary size — none of which register in a tok/s figure.

The `@Tool("Description")` UDA approach has a structural implication worth noting. In D, user-defined attributes are resolved at compile time, meaning tool discovery happens during compilation rather than at agent startup. This is architecturally link-time registration, not runtime reflection. For the Docker-sandboxed execution tool specifically, the sandbox entry points are statically known and auditable — relevant for reproducible builds, though not a substitute for sandbox escape analysis.

## What cannot be concluded

DLLM does not publish which specific models populate its three-model setup. The 8 GB figure fits several combinations — a 7B agent with a compact summary and a sub-1 GB embed model, or three smaller models with a generous KV-cache allocation. The table above bounds the feasible space from our measured GGUF sizes; it does not identify which combination DLLM's author actually uses. The importC assessment covers only call-dispatch and runtime overhead — prefill latency, CUDA stream contention across three concurrent allocations, and the throughput cost of KV-cache condensation passes are outside its scope.

## Thumbdash

Thumbdash runs SmolLM2-360M-Instruct through llama.cpp on Metal — a single model whose entire weight fits inside the KV-cache residual from even the most constrained pipeline combination in the table above. The two projects target different problems. DLLM optimises for context-window capacity and multi-model coordination on a CUDA device; Thumbdash optimises for sub-100 ms keystroke-response latency on-device, with no VRAM contention between models. DLLM's `@Tool` compile-time registration and KV-cache condensation have no direct application in Thumbdash's single-turn loop, but where multi-model pipeline complexity earns its cost is covered in [our notes on agentic DSP verification](/blog/auralinter-agentic-dsp-verification).

---

*Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of [DannyArends/DLLM](https://github.com/DannyArends/DLLM) and this studio's own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (10 claim(s) and 1 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (9 derived from it, 10 row(s) supplied from our own tables). The studio has not re-run DannyArends/DLLM's benchmarks; figures attributed to it are its own.*

## Questions this note answers

### Does importC actually make DLLM faster than a Python-binding equivalent?

Not in tokens per second. On a CUDA device, inference throughput is bounded by VRAM bandwidth and model weight volume — importC changes neither. The advantage is deployment-graph simplicity: no Python interpreter, no binding layer to version-pin, no serialisation overhead between the D process and llama.cpp. Cold-start time and binary footprint are different; those improve. Sustained decode rate does not.

### Are all three models resident in VRAM simultaneously?

DLLM claims 8 GB VRAM for the full described setup alongside CUDA offloading. If models were swapped in and out per pipeline stage, the 8 GB figure would instead describe peak single-model VRAM — a much weaker and nearly uninformative constraint. The natural reading of "the above model setup requires 8 GB VRAM" is that all three weight sets are resident concurrently. That is the assumption the weight-budget analysis uses.

### What decode rate should I expect for the agent model on an 8 GB VRAM card?

Decode rate scales with VRAM bandwidth, and llama.cpp consistently saturates it. An 8 GB consumer card is not the RTX 4090 at 1,008 GB/s, so the 240 tok/s figure for 7B Q4 on that device does not apply directly. What does hold: KV-cache condensation passes — which re-run part of the model to compress context — consume bandwidth from the same budget as decode. Condensation frequency is therefore a direct lever on effective throughput on bandwidth-limited hardware.

### Does KV-cache condensation change the weight-budget analysis?

No. KV-cache condensation reduces the KV-cache footprint at runtime by evicting or summarising stored key-value pairs; it does not touch the loaded model weights W_i for any of the three models. Condensation shrinks the KV_budget term in the inequality — useful for sustaining longer conversations within a fixed VRAM ceiling — but the weight allocation is unchanged.