ANEForge hits 2.28× speculative decode

ANEForge’s 2.28× speculative-decode speedup (7.4 → 16.8 tok/s on Qwen3-8B) is the ceiling the verify(K) ≈ verify(1) condition allows. Nothing is left to recover on the verification side. Work backward from those figures under int4-LUT compression and the baseline implies 29.6 GB/s effective ANE bandwidth — below the 42.8 GB/s our published Orion measurement implies for GPT-2 124M on a split ANE+CPU path, consistent with an 8B model streaming from unified DRAM rather than on-chip SRAM.

What ANEForge claims

ANEForge states it compiles tensor graphs directly into ANE programs through the private aned stack, bypassing CoreML entirely. The claims below are reported by ANEForge; the studio has not independently verified them.

Claim	Figure	Conditions
ResNet-18 inference	0.33 ms	Pretrained, cosine 1.0000 vs. reference, pure ANE
MLPerf ResNet-50 accuracy	fp16 76.44% (= fp32)	submission_checker v5.1, all 3 edge scenarios VALID
Speculative decode	2.28× (7.4 → 16.8 tok/s)	Qwen3-8B + 0.6B draft, exact speculation
Retrieval embedding	4–5× faster than GPU	cosine 1.0000, LangChain Embeddings drop-in
Energy efficiency	8–16× better than GPU	All three measured workloads; GPU leads ViT-B/16 in latency
Stable Diffusion 1.5	U-Net 1.5%, VAE 4.4% rel. error	Per-component vs. reference
Fluid simulation energy	~9 J total	1.48 W measured rail
Dispatch floor	~70 µs	Program reused across calls
int4-LUT compression	~4× smaller	vs. fp16; dequant on-engine
Cross-compilation	28 ANE targets (M1–M5)	Latency estimation included; no target hardware required

ANEForge also states that af.sdpa exposes the fused-attention layer Apple’s compiler decomposes and never emits, and that 18 additional native layers (argmax, topk, sort, geometry) are accessible that CoreML cannot reach.

Bandwidth-bound decode and the Orion denominator

Autoregressive decode is bounded by how fast model weights stream from memory per generated token. The governing relationship is:

$$\text{tok/s} = \frac{B_\text{eff}}{M_\text{model}}$$

where $B_\text{eff}$ is effective memory bandwidth in bytes per second and $M_\text{model}$ is model size in bytes. Our Orion field note measured 172.4 tok/s for GPT-2 124M fp16 with transformer blocks on ANE and wte + logits on CPU. That model occupies $124 \times 10^6 \times 2 = 248\,\text{MB}$, so:

$$B_\text{eff,\,Orion} = 172.4 \times 0.248\,\text{GB} = 42.8\,\text{GB/s}$$

Because wte and logits ran on CPU in that measurement, the ANE-only path exceeded this — 42.8 GB/s is a conservative floor.

ANEForge reports Qwen3-8B at 7.4 tok/s baseline. ANEForge’s int4-LUT claim puts the compressed model at $(8 \times 10^9 \times 2)\,/\,4 = 4\,\text{GB}$. Substituting:

$$B_\text{eff,\,base} = 7.4 \times 4\,\text{GB} = 29.6\,\text{GB/s}$$

With speculative decode at 16.8 tok/s under the same int4 assumption:

$$B_\text{eff,\,spec} = 16.8 \times 4\,\text{GB} = 67.2\,\text{GB/s}$$

Model	Format	$M_\text{model}$	tok/s	Implied $B_\text{eff}$	Path	Source
GPT-2 124M	fp16	0.248 GB	172.4	42.8 GB/s	ANE + CPU (split)	Our Orion measurement
Qwen3-8B	int4 (derived from claim)	4 GB	7.4 (base)	29.6 GB/s	ANE	ANEForge reports
Qwen3-8B	int4 (derived from claim)	4 GB	16.8 (spec)	67.2 GB/s	ANE	ANEForge reports

The baseline Qwen3-8B decode (29.6 GB/s) sits below Orion’s conservative floor (42.8 GB/s). GPT-2 124M at 248 MB fits in ANE on-chip SRAM; Qwen3-8B at 4 GB (int4) does not, so every token incurs DRAM latency — which explains the bandwidth shortfall despite comparable hardware. The speculative case (67.2 GB/s) exceeds the Orion floor, suggesting verification’s batch structure loads the memory bus more efficiently than single-token decode even when weights reside in DRAM.

Speculative decode ceiling

When verify(K) ≈ verify(1), the per-round cost is dominated by one target model pass regardless of how many draft tokens are being verified. The speedup formula collapses to:

$$\text{speedup} = 1 + \bar{k}$$

where $\bar{k}$ is the expected accepted draft tokens per round. Substituting ANEForge’s figures:

$$\bar{k} = \frac{16.8}{7.4} - 1 = 2.28 - 1 = 1.28\,\text{tokens per round}$$

That figure is both the measured speedup and the theoretical ceiling under the stated condition. Exceeding it without changing verify behavior requires either a higher-quality draft model (raising $\bar{k}$) or a faster target decode (raising the 7.4 tok/s base). ANEForge has extracted everything the verification path can offer.

On GPU, verify(K) typically grows with K once the batch becomes compute-bound, forcing a tradeoff between draft length and verification cost. ANE sidesteps this. Autoregressive decode on ANE is memory-bandwidth-bound rather than compute-bound — the same property that set the bandwidth floor above — so verify(K) ≈ verify(1) holds. ANEForge’s claim that “decode is latency-bound” is the mechanistic explanation for why the simplified speedup formula applies here and not on GPU.

Dispatch floor and audio buffer compatibility

ANEForge reports a ~70 µs dispatch floor for reused programs. At 48 kHz with a 256-sample buffer, the available window per block is:

$$T_\text{block} = \frac{256}{48000} \approx 5.33\,\text{ms}$$

The dispatch overhead consumes $70\,\mu\text{s}\,/\,5330\,\mu\text{s} \approx 1.3\%$ of that budget. At 128 samples (2.67 ms) it rises to 2.6%. Adding ResNet-18 at ANEForge’s reported 0.33 ms brings total ANE overhead to $0.07 + 0.33 = 0.40\,\text{ms}$, or 7.5% of a 5.33 ms window — a workable margin before remaining DSP budget is allocated.

Compile amortization is a separate cost. ANEForge states programs compile once and reuse, but publishes no compile times. Our Orion note measured ~4,200 ms full compile for Stories110M — the closest published proxy. If ANEForge’s times are in that range, break-even at the dispatch floor requires:

$$N_\text{break} = \frac{4200\,\text{ms}}{0.070\,\text{ms}} = 60{,}000\,\text{dispatches}$$

At 187.5 dispatches per second (256-sample / 48 kHz), that is 320 seconds of sustained runtime. First-launch compile latency matters for a live session. For offline batch neural DSP, it’s a rounding error.

What cannot be concluded

ANEForge does not state the weight format for the Qwen3-8B benchmark. The bandwidth analysis treats the int4-LUT compression claim as the operative assumption; if the benchmark ran at a different quantization level, all three $B_\text{eff}$ figures shift proportionally. At fp16, the baseline would imply $7.4 \times 16\,\text{GB} = 118.4\,\text{GB/s}$ — hard to reconcile with Orion’s 42.8 GB/s floor without a separate explanation. The compile amortization calculation borrows Orion’s figure for a different model and graph structure; it establishes a directional bound, not a precise prediction for ANEForge’s 58-fused-operator compilation path.

What this means for Biquadia

The 70 µs dispatch floor puts ANE-backed neural DSP inside real-time audio budgets at 256-sample / 48 kHz — under 8% overhead for a ResNet-18-scale model. For Biquadia’s always-on neural enhancement path, the 8–16× energy advantage ANEForge reports over GPU is directly relevant: at the 1.48 W rail ANEForge measured for its fluid simulation, sustained inference is thermally viable on passively cooled M-series silicon. GPU use is not.

The 18 native layers unreachable through CoreML — including argmax, topk, and sort — and the fused-attention path matter for any Biquadia model component that currently falls back to CPU because CoreML decomposes or rejects the op. Cross-compilation support across all 28 targets (M1–M5) reduces per-device test matrices without requiring every chip generation in-house. For latency budget context on the ANE audio path, see our presspeech note; for ANE compile overhead precedent, see our Orion decode analysis.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from two sources — the published README of sbryngelson/ANEForge and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (18 claim(s) and 0 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (21 derived from it, 19 row(s) supplied from our own tables). The studio has not re-run sbryngelson/ANEForge’s benchmarks; figures attributed to it are its own.

FAQ

Is the private API dependency a shipping risk?

ANEForge states it operates from an ordinary user process with no entitlements and without disabling SIP. The aned stack it calls is also used by CoreML, MPSGraph, and Espresso — Apple has strong incentive not to break it silently. That said, Apple has changed private ANE interfaces across OS point releases before, and ANEForge makes no ABI stability claim. Any Biquadia integration path would need a functioning CoreML or CPU fallback.

Does passing the MLPerf submission_checker mean this is production-grade inference?

ANEForge reports passing MLCommons submission_checker v5.1 for all three edge scenarios with fp16 accuracy equal to fp32 (76.44%). The checker validates output accuracy against a reference implementation. It does not cover sustained-load throughput, thermal throttling, or multi-session isolation. The VALID result is a meaningful accuracy bar, not a substitute for a long-duration inference profile under real workload conditions.

Why is the Qwen3-8B baseline (7.4 tok/s) so much slower than Orion's GPT-2 result (172.4 tok/s)?

The models differ by roughly 64× in parameter count. The bandwidth analysis shows the 8B baseline implies only 29.6 GB/s effective bandwidth versus 42.8 GB/s for GPT-2 124M. GPT-2 124M at 248 MB sits in on-chip memory; Qwen3-8B at 4 GB (int4) does not, and the bandwidth shortfall reflects that — DRAM access latency on every token.

What compile overhead should I budget when adopting ANEForge?

ANEForge does not publish per-model compile times — only the ~70 µs per-call dispatch floor for reused programs. Our Orion note measured ~4,200 ms full compile for Stories110M, the closest available proxy for ANE program compilation at similar scale. At 187.5 dispatches per second under a 256-sample / 48 kHz audio workload, amortizing a 4,200 ms compile to under 1% of per-call cost requires roughly 320 seconds of operation — meaningful for first-launch user experience, negligible for offline batch pipelines.