# Orion's ANE Decode vs CPU Baseline

Driving Apple's Neural Engine through private frameworks: 172.4 tok/s on ANE vs 283 on CPU, and delta compilation that pays for itself in 1.21 steps.

Canonical page: https://makerportal.ai/blog/biquadia-github-1171688936
Author: Joshua Hrisko, Principal Engineer — MakerPortal
Published: 2026-08-05
Section: Field note / On-Device AI · 7 min read
Tags: biquadia, on-device-ai, metal-ane

---

Orion's own figures show ANE decode at 172.4 tok/s trails its CPU baseline of 283 tok/s by 39% on GPT-2 124M, meaning the accelerator path is currently slower than the fallback for inference on this model class. The more defensible result is the delta compilation mechanism, which amortises its startup cost after just 1.21 training steps.

## What Orion claims, with conditions attached

Orion reports two headline numbers under different conditions and they should not be conflated.

| Measurement | Figure | Model | Hardware path | Conditions |
|---|---|---|---|---|
| ANE inference speed | 172.4 tok/s (= 1 ÷ 5.8 ms) | GPT-2 124M fp16 | ANE (transformer blocks) + CPU (wte, 73 MB logits) | Orion v1.0, Apple Silicon, private ANE APIs |
| CPU baseline | 283 tok/s | GPT-2 124M fp16 | CPU only, full forward pass | M1, Orion CPU milestone |
| Delta reload time | ~494 ms | Stories110M | ANE, 60 kernels | 72 programs pre-compiled at startup (~4.5 s) |
| Full recompile time | ~4,200 ms | Stories110M | ANE | Per-step, no delta path |
| 1,000-step wall time (delta) | 22 min | Stories110M | ANE | Loss 12.3→8.9, zero NaN reported |
| 1,000-step wall time (full recompile) | ~85 min | Stories110M | ANE | Orion's projection, not a separate measured run |

Orion also states the ANE delivers approximately 19 TFLOPS fp16, citing a third-party analysis that Apple's official 38 TOPS INT8 rating involves dequantisation to fp16 before computation.

## Analysis

**Delta compilation amortisation**

Define total wall time for N training steps as:

$$
T_{\text{delta}}(N) = T_{\text{start}} + N \cdot (T_{\text{reload}} + T_{\text{compute}})
$$

$$
T_{\text{full}}(N) = N \cdot (T_{\text{fullcompile}} + T_{\text{compute}})
$$

The break-even step count N\* where delta becomes cheaper:

$$
N^* = \frac{T_{\text{start}}}{T_{\text{fullcompile}} - T_{\text{reload}}} = \frac{4.5\text{ s}}{4.2\text{ s} - 0.494\text{ s}} = \frac{4.5}{3.706} \approx 1.21 \text{ steps}
$$

Delta compilation wins from step 2 onward. At 1,000 steps the gross per-step saving is 4.2 - 0.494 = 3.706 s/step, accumulating to 3,706 s of avoided compilation less the 4.5 s startup cost — a net saving of 3,701.5 s ≈ 61.7 minutes. Orion reports observed savings of 85 - 22 = 63 minutes, consistent within the variance in T_compute across the two paths.

From the 22-minute wall time, the average total step time is 22 × 60 ÷ 1,000 = 1.32 s. Subtracting the 494 ms reload leaves a per-step compute residual of 0.826 s — that is the forward pass, backward pass, and optimiser step on the ANE. The 8.5× recompilation speedup and the 3.8× total step speedup are not contradictory: 8.5× describes only the compile/reload ratio (4,200 ÷ 494 = 8.50×), while 3.8× describes total step time once the 0.826 s compute floor is included. The ~119 compile-per-process limit Orion documents is bypassed entirely because the 72 startup programs remain resident; 1,000 training steps make 1,000 reload calls but zero additional compile calls.

**Effective ANE bandwidth from the decode figure**

GPT-2 124M at fp16 occupies 124 × 10⁶ × 2 = 248 MB. Orion runs the wte / logits projection on CPU because the 73 MB embedding table exceeds ANE SRAM, leaving 248 - 73 = 175 MB for the ANE. At batch size 1, transformer decode is bandwidth-bound (arithmetic intensity ≈ 1 FLOP/byte), so the decode latency directly bounds effective bandwidth:

$$
B_{\text{eff}} = \frac{W_{\text{ANE}}}{t_{\text{tok}}} = \frac{(248 - 73)\text{ MB}}{5.8 \times 10^{-3}\text{ s}} = \frac{175\text{ MB}}{0.0058\text{ s}} \approx 30.2\text{ GB/s}
$$

For comparison, the studio's published reference data shows an M4 Max (rated 400 GB/s LPDDR5X) reaching approximately 220 tok/s on a 3B Q4 model — an effective bandwidth of 1,800 MB × 220 tok/s = 396 GB/s, near the rated ceiling (see [Running 3B LLMs on edge SBCs](/blog/running-3b-llms-on-microcontrollers)). The ANE's 30.2 GB/s is not caused by a memory-bandwidth-starved platform; it reflects what the ANE delivers to a single batch-1 decode stream on this model. The compute ceiling (19 TFLOPS) is never binding here: sustaining that throughput at ~1 FLOP/byte arithmetic intensity would require ~19 TB/s of memory bandwidth, a physical impossibility.

**Why ANE decode trails CPU on this model**

Orion's CPU baseline of 283 tok/s for GPT-2 124M exceeds the ANE decode figure of 172.4 tok/s by a factor of 1.64×. The split execution forces every token to cross the ANE/CPU boundary twice — once for the embedding lookup and once for the tied output projection — on a model small enough that the IOSurface handoff and fp16 [1, C, 1, S] tensor layout conversion overhead dominates. Orion does not claim ANE inference is faster than CPU inference for this configuration; the "170+ tok/s" headline is positioned against larger or slower external configurations rather than Orion's own CPU baseline.

**RSS growth and the endurance claim**

The M4 weight-swapping milestone reports RSS growth of 1.41× over 100 swap iterations. The 1,000-step endurance run reports "no memory leak," but these are measurements from different milestone phases and their hardware and runtime conditions are not stated to be identical. The tension between 1.41× growth at 100 iterations and a clean bill at 1,000 steps is not resolved in the published material.

## What cannot be concluded

The private API dependency is the single hardest constraint on reproducibility and longevity. Orion targets `_ANEClient` and `_ANECompiler` directly — neither is documented by Apple, and both can change silently across macOS minor versions. The delta compilation path, the LoRA hot-swap via IOSurface inputs, and the compile-limit bypass all sit on this foundation. The 14 newly documented ANE constraints Orion reports are empirical observations of hardware behaviour at the time of measurement, not guaranteed invariants. The ~85-minute full-recompile comparison is also Orion's own projection, not a separately measured run.

## What this means for Biquadia

Biquadia already targets Apple Silicon for real-time neural DSP. The pertinent result from Orion is not the inference throughput — CPU-side processing outperforms the split-ANE path at 124M-parameter scale — but the delta compilation pattern. Neural DSP workloads share exactly the structural property that governs the 1.21-step break-even: programs are compiled once per session, weights update frequently, and session lengths are short. A typical plugin session runs tens of minutes, well past the single-step amortisation point.

Any ANE-targeted enhancement model in Biquadia that updates weights at runtime — adaptive noise suppression, per-session model personalisation, dynamic spectral processing — would see the same compile-overhead profile. The 4.5 s startup compiles once per session; the 494 ms reload sits well below the latency budget of a professional audio buffer cycle at typical block sizes. The split-execution constraint has a direct analogue here too: any weight tensor that exceeds available ANE SRAM forces a CPU fallback, incurring the same synchronisation cost that Orion's own numbers make visible at 124M-parameter scale.

For the broader picture of on-device inference density relative to memory bandwidth, see [Running 3B LLMs on edge SBCs](/blog/running-3b-llms-on-microcontrollers) and [what it takes to run semantic search on-device](/blog/notiary-on-device-semantic-search).

## Questions this note answers

### Does the 8.5× figure contradict the 3.8× figure?

No. The 8.5× is the ratio of per-step compile time to reload time: 4,200 ÷ 494 = 8.50×. The 3.8× is the ratio of total step time, which Orion's wall-time figures imply includes approximately 826 ms of compute per step beyond the reload. Once that compute floor is included, the savings from faster reloading are diluted and the total-step ratio falls to 3.8×.

### Why does the CPU baseline beat ANE decode for GPT-2 124M?

Orion splits execution across two domains: transformer blocks on the ANE, the 73 MB wte embedding table on CPU. At 124M-parameter scale the CPU–ANE synchronisation overhead — IOSurface handoffs, fp16 [1, C, 1, S] layout conversion — consumes enough of the available latency budget that pure CPU wins. The ANE advantage for inference would likely appear on a model where the full weight set fits within ANE SRAM, or on a batched workload where arithmetic intensity rises above the bandwidth-bound regime.

### Is the 1,000-step training result sufficient to assess gradient stability?

The reported loss decrease from 12.3 to 8.9 with zero NaN is consistent with a functioning training loop. However, 1,000 steps on Stories110M covers only the initial loss regime and does not exercise learning rate schedule edge cases, gradient clipping saturation, or divergence modes that appear at longer horizons. The RSS growth of 1.41× observed at 100 swap iterations in a separate milestone also leaves open whether memory usage plateaus or continues growing under extended training.

### Does Orion require any official Apple APIs?

Orion reports that it bypasses CoreML entirely and calls `_ANEClient` and `_ANECompiler` directly, both private frameworks. The build uses a Makefile across approximately 50 source files and requires no Xcode project. Private APIs carry no Apple stability guarantee; a macOS point release can alter their behaviour or remove them without notice, which makes the compile-limit bypass and delta reload mechanism fragile across OS updates.