Core ML ANE's 2.78× energy advantage over MLX is really 2.0× in a sustained loop

The 2.78× energy-per-decision advantage that laya-coreml reports for Core ML ANE over compiled MLX is not a single effect. It decomposes into a ~2.0× system power ratio and a 1.39× inference speed ratio. In a sustained loop where the device is continuously active, only the first term contributes to actual energy savings. The effective energy advantage in a continuous workload is closer to 2.0× than 2.78×.

What laya-coreml is

Laya-CoreML is a typed decision model, not a generative LLM. It selects from a fixed set of options given a short prompt and runs on Apple Silicon via Core ML. A dedicated Neural Engine bundle enforces a 96-token hard limit on total input (question, options, and state combined); the general-purpose GPU variant supports 1024 tokens. The intended use case is bounded, low-latency classification — the Snake game loop in the repo is the reference application.

The figures and their conditions

All measurements are on M3 Max (40-core GPU, 128 GiB, macOS 27.2), one 91-token question padded to 96 tokens, synchronous inference including prompt preparation, tokenization, array construction, calibration, and formatting. Six alternating 20-second blocks per implementation yielded 65,598 stable calls total.

Metric	Compiled MLX FP16	Core ML ANE FP16	Core ML ANE W8
P50 / P95 latency	6.94 / 7.39 ms	4.98 / 5.31 ms	4.88 / 5.23 ms
Mean system power	61.39 W	30.75 W	27.39 W
System energy / decision	0.4288 J	0.1540 J	0.1344 J
Speed gain vs MLX	1×	1.39×	1.42×
Energy gain vs MLX	1×	2.78×	3.19×

The ANE FP16 variant passes 59/59 fitting questions with maximum calibrated-probability drift of 0.002925. The W8 variant passes the same subset with drift 0.014393 under a 0.02 gate. The 1024-token GPU variant takes approximately 91.7 ms per request in its serial screen.

Reported by mizorewww/laya-coreml tokens· scroll →

Conditions around the comparison

scroll →

49.1–50.0 decisions/s

sustained decision rate in the complete active Snake

4.98 ms

median latency for one short multilingual

2.78×

whole-system energy

4.88 ms

W8 variant latency

Decomposing the energy ratio

The energy-per-decision figure is a product of two independent quantities:

$$E_{\text{decision}} = P_{\text{system}} \times t_{\text{inference}}$$

The ratio between the MLX and ANE FP16 implementations is therefore:

$$\frac{E_{\text{MLX}}}{E_{\text{ANE}}} = \frac{P_{\text{MLX}}}{P_{\text{ANE}}} \times \frac{t_{\text{MLX}}}{t_{\text{ANE}}}$$

Substituting the reported values:

$$= \frac{61.39}{30.75} \times \frac{6.94}{4.98} = 1.9964 \times 1.3936 = 2.782$$

The two factors contribute unequally to the logarithm of the ratio:

$$\ln(1.9964) = 0.691, \quad \ln(1.3936) = 0.332, \quad \text{total} = 1.023$$

The power ratio accounts for 67.6% of the log-energy-gain. The speed ratio gets the remaining 32.4%.

They matter under different operating conditions. The speed ratio ($t_{\text{MLX}} / t_{\text{ANE}}$) reduces energy per event when the system can idle between inferences — shorter inference means more time in a low-power state. The power ratio ($P_{\text{MLX}} / P_{\text{ANE}}$) is relevant whenever the system is active, regardless of inference duration.

In the sustained Snake loop — 49.1–50.0 decisions/s, no idle gaps — the system draws power continuously. Energy per unit wall-clock time is simply the power draw. Inference speed sets decisions per second but does not reduce the watts drawn during those seconds. The relevant comparison here is 61.39 W versus 30.75 W. A factor of 1.996, not 2.78.

The 2.78× figure is correct as a burst metric: make one decision, sleep the device, repeat. For a real-time loop that never sleeps, the ceiling on energy savings is the power ratio alone.

What cannot be concluded

The sustained Snake loop rate of 49.1–50.0 decisions/s is reported for the ANE implementation. The source does not state what rate the MLX implementation sustains in the same loop. Without that figure, it is impossible to determine whether the MLX path would also sustain ~50 decisions/s (loop bottlenecked by game logic, power ratio is the entire story) or a lower rate (speed component still partially matters, because the GPU path would spend part of each frame in a higher-power state waiting for inference to complete).

What this means for a sustained on-device loop

For an engineer building a continuous inference pipeline on Apple Silicon — a real-time DSP chain, a game loop, a sensor fusion system — the decision between ANE and GPU should be made on power draw, not per-event energy. The ANE’s 30.75 W versus the GPU’s 61.39 W is a 50% reduction in system power during active inference.

The 1.39× speed advantage is secondary in this regime. It matters if your loop has a hard deadline that the GPU path would miss but the ANE path would meet. At ~5 ms per inference against a 20 ms frame budget (50 fps), both paths meet the deadline comfortably. The speed factor does not change the architectural choice.

For Biquadia, whose neural enhancement models run in a continuous real-time audio loop, the relevant question when choosing between ANE and GPU for a new model is: what is the system power draw during active inference? The per-event energy figure from a burst benchmark overstates the thermal benefit by a factor of 1.39 in a loop that never idles. The pattern is the same one that showed up in the Bonsai 2 27B throughput analysis: headline ratios that multiply independent effects look larger than any single engineering decision can actually capture.

---

Method: this note was drafted by qwen/qwen3.8-27b from two sources — the published README of mizorewww/laya-coreml and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (34 claim(s) and 35 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (14 derived from it, 47 row(s) supplied from our own tables). The studio has not re-run mizorewww/laya-coreml’s benchmarks; figures attributed to it are its own.

FAQ

Does the W8 variant change the decomposition?

The W8 energy gain is 3.19×, which decomposes as (61.39/27.39) × (6.94/4.88) = 2.241 × 1.422 = 3.187. The power ratio grows to 2.24× and the speed ratio to 1.42×. The same logic applies: in a sustained loop the relevant figure is 2.24×, not 3.19×.

Can the 96-token limit be worked around for longer inputs?

The source states the ANE bundle has a 96-token total limit including question, options, and state. The 1024-token variant runs on CPU + GPU, not ANE. No ANE variant with a higher token limit is stated. If your decision prompt exceeds 96 tokens, you must use the GPU path and accept the 61.39 W power draw.

Is the 4.98 ms figure inclusive of everything an application would pay?

The source states the measurement includes "prompt preparation, tokenization, arrays, synchronous inference, calibration and formatting." It does not include first-load model allocation, inter-process communication, or framework dispatch latency. The 4.98 ms is the steady-state per-call cost after the model is loaded and warm.

Why does the ANE draw less system power than the GPU for the same model?

The source does not explain the mechanism. It reports 30.75 W for ANE FP16 versus 61.39 W for compiled MLX FP16 on the same M3 Max, without attributing the gap to a specific cause. The difference could reflect the ANE's dedicated datapath, reduced memory bandwidth for a 322M-parameter model, or differences in how the two runtimes manage SoC power domains.