Espresso bounds CoreML overhead at 168.7 µs

The 3.41× speedup Espresso claims over CoreML on its 6-layer benchmark artifact, read against our published ~70 µs ANE dispatch floor, bounds the CoreML-attributable per-token overhead at a minimum of 168.7 µs for this artifact class. Both greedy divergences on the Qwen2.5-0.5B path land on the runner-up token — margins of 0.027 and 0.069 logits. That puts the fp16 ANE noise at the decision boundary rather than uniformly distributed, which changes what the 11.2% token divergence rate actually implies.

What Espresso claims

Espresso states it compiles MIL programs to ANE silicon through reverse-engineered private APIs (_ANEClient, _ANEInMemoryModel) with no CoreML in the hot path and no per-token recompilation. The benchmark data it publishes — CI-enforced against latest.json — covers two distinct artifacts:

Measurement	Value	Artifact	Execution path	Conditions
Logit MAE, CPU probe	9.3 × 10⁻⁵	Qwen2.5-0.5B-Instruct	fp32 CPU	All 24 layers; NumPy LM head, not served classifier
Logit agreement, ANE probe	~0.96	Qwen2.5-0.5B-Instruct	ANE hybrid	All 24 layers; NumPy LM head, not served classifier
Sequences token-for-token	10 / 12	Qwen2.5-0.5B-Instruct	ANE hybrid, greedy	Fixed 12-prompt suite
Token agreement	341 / 384	Qwen2.5-0.5B-Instruct	ANE hybrid, greedy	384 tokens across 12 prompts
Speedup vs CoreML	3.41×	6-layer demo artifact	ANE fused recurrent	dim=768, 12 heads, 32k vocab, seqLen=256, M3 Max, macOS 15

The first four rows come from the Qwen path. The speedup row comes from a separate, shallower demo artifact. These two artifacts do not share architecture.

Analysis

Bounding CoreML per-token overhead

Our published ANE dispatch floor — ~70 µs for a program reused across calls — provides a denominator for the 3.41× speedup. Model per-token latency for each path as the sum of a hardware dispatch component $D$ and a compute component $T_c$, with CoreML adding an additional overhead $\Delta$ above the hardware floor:

$$S = \frac{D + \Delta + T_c}{D + T_c} = 3.41$$

Under the conservative assumption that $T_c$ is identical for both paths — overhead removal only, no kernel fusion benefit — this gives:

$$\Delta_{\min} = (S - 1) \cdot D = (3.41 - 1) \times 70\,\mu\text{s} = 2.41 \times 70\,\mu\text{s} = 168.7\,\mu\text{s}$$

This is a lower bound. If Espresso’s fused recurrent kernels also reduce $T_c$, the bound only tightens: actual CoreML overhead must be larger, not smaller, because Espresso would then be outperforming the pure overhead-elimination model. So 168.7 µs is the weakest reading the 3.41× ratio supports.

Two constraints cap how far this extrapolates. The 6-layer artifact is very shallow — with $T_c$ small relative to $D$, the overhead fraction is amplified and produces an unusually large headline speedup that compresses at production model depth. Separately, Espresso characterises “up to ~1 logit” of ANE fp16 noise from the Qwen probe, not from the 6-layer artifact; these figures belong to different measurement contexts.

Fidelity structure of the divergences

Across the 12-prompt greedy suite, 341 of 384 tokens match the PyTorch fp32 reference: 341/384 = 88.8% agreement. The divergence count is (384 − 341)/384 = 43/384 ≈ 11.2% of individual token decisions. At sequence level the picture sharpens: 10 of 12 sequences are fully clean, so all 43 divergent tokens concentrate in 2 sequences rather than distributing across 12.

Where those failures land matters. Espresso reports both divergences fall on the reference’s second-ranked token, at top-1/top-2 logit gaps of 0.027 and 0.069. The fp16 ANE path introduces up to ~1 logit of absolute error — a budget roughly 37× the smaller gap and 14× the larger one. Tokens where the margin between first and second rank exceeds ~1 logit are immune to this error mode under greedy decoding: no noise of that magnitude can perform a rank swap at a margin of, say, 2 logits. These aren’t random fp16 bit failures. They’re minimum-energy failures at the decision boundary — a categorically different error mode from uniform noise.

The CPU probe result — 9.3 × 10⁻⁵ MAE across all 24 layers — establishes that Espresso’s MIL compilation is arithmetically faithful in fp32. The gap between that figure and the ~0.96 ANE hybrid agreement is entirely attributable to fp16 execution, not to graph transformation errors. The two metrics are not on a common scale: 9.3 × 10⁻⁵ is an absolute MAE in logit units; ~0.96 is a normalized agreement score (likely cosine similarity or Pearson correlation). Espresso states this directly.

What cannot be concluded

The 3.41× speedup and the fidelity measurements live on different artifacts and cannot be composed. Speedup is from a 6-layer demo; token accuracy is from a 24-layer production model. Nothing in Espresso’s published data characterises speedup at production depth, nor fidelity of the 6-layer demo on any task. The private API surface carries no stability guarantee: _ANEClient and _ANEInMemoryModel are undocumented symbols that can be renamed or removed in any macOS security update, with failures appearing only at runtime. Neither the studio nor this note has run Espresso.

What this means for Biquadia

Biquadia’s neural enhancement models run fixed compute graphs at inference time — exactly the scenario where our ~70 µs dispatch floor applies and where the 168.7 µs lower bound on CoreML overhead is the relevant latency signal. At a 512-sample frame size and 44.1 kHz, audio inference runs at roughly 86 Hz. That’s well below any dispatch-floor throughput ceiling; the concern is per-call latency that adds directly to frame budget, not throughput per se.

The fidelity results bear on Biquadia differently than on language generation. Enhancement and spatial audio models output continuous activations — filter coefficients, gain curves, spatial parameters — rather than ranked discrete tokens. The margin-dependent divergence mechanism Espresso’s data reveals requires a rank-inversion boundary to flip. Regression or classification heads with typical output distributions rarely sit at margins of 0.027 logits. The 11.2% token divergence rate is a structural worst case for this error mode applied to language generation; it’s not a useful prior for Biquadia’s task types.

The App Store constraint is a hard stop for any distributed build. The efficiency and overhead advantages the data supports are accessible without private APIs via CoreML’s programmatic ANE routing, at the cost of the overhead this note quantifies. For ANE throughput at production model depth, see ANEForge hits 2.28× speculative decode; for on-device embedding workloads operating under this same dispatch budget, see on-device vector search in Swift.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of christopherkarani/Espresso and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (11 claim(s) and 6 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (11 derived from it, 16 row(s) supplied from our own tables). The studio has not re-run christopherkarani/Espresso’s benchmarks; figures attributed to it are its own.

FAQ

Why does 88.8% token agreement coexist with only 2 failing sequences?

Espresso reports 10 of 12 sequences as fully token-for-token correct. The 43 divergent tokens are concentrated in the 2 failing sequences; the other 10 contribute zero divergences each. When failures cluster in a small number of prompts with small logit margins rather than distributing uniformly, sequence-level pass rate looks substantially better than raw token accuracy — expected, given that divergences require the margin to fall within the ~1 logit noise budget.

Are the 9.3 × 10⁻⁵ and ~0.96 agreement figures measuring the same thing?

No. The 9.3 × 10⁻⁵ figure is a mean absolute error of logit values on the fp32 CPU path, measuring arithmetic fidelity of Espresso's MIL compilation. The ~0.96 figure is a normalized agreement score on the ANE hybrid path's per-layer outputs. Different metrics, different execution paths — no conversion makes them directly comparable on a common scale, and Espresso does not claim otherwise.

Does the 3.41× speedup hold at production model depth?

Almost certainly not at that magnitude. Production models have substantially more compute per token relative to the dispatch floor, which shrinks the overhead fraction and compresses the headline speedup ratio even if the absolute overhead saving stays the same. Espresso has not published speedup measurements for any model deeper than its 6-layer demo artifact, so there is no published basis for extrapolating the figure to 24-layer or larger architectures.

What breaks first when a macOS update ships?

The private symbols _ANEClient and _ANEInMemoryModel and the IOSurface buffer layout are undocumented and version-specific. Espresso's CI enforces README/measurement consistency but cannot gate on Apple ABI changes. Any macOS update that restructures or renames these symbols breaks the dispatch path silently — no compiler error, no linker warning — with failure surfacing only at the first runtime call to the ANE.