ANE routes by expression, not arithmetic

The study establishes that ANE placement is governed by how a computation is expressed rather than what it computes, and that decode throughput follows a bytes-per-token bandwidth law (~0.77 × nominal encoding width) that holds across fp16, int8, and 2-bit precisions. What it does not establish: whether models optimised for ANE residency retain task accuracy relative to their fp16 counterparts at any of the tested parameter counts.

Study design

Three measurement paths. A 64-shape matrix of LLM primitives varied the expression of each operation while holding its arithmetic output fixed, recording per-operation device placement. Matched models were trained across size and precision; their quantized checkpoints are byte-identical in structure to the fp16 counterparts, so placement comparisons don’t carry confounds from graph restructuring. The third path is the ground truth: ANE memory-controller byte counters read during inference, capturing what actually executed on the engine rather than what the CoreML compiler reported as intended placement.

What was measured	Compared against	Sample	Conditions
Per-operation ANE device placement	Arithmetically identical alternative expressions of the same operation	64 expression shapes across LLM primitives	CoreML compiler; specific Apple chip generation not stated in source
ANE residency (% bytes through engine) and decode speed (ms/token)	Conv-heavy fp16 design at 25.85M parameters	22.29M all-attention fp16; 25.85M conv-heavy in fp16, int8, 2-bit; 25M and 50M ternary variants	ANE memory-controller byte counters read during inference; serving batch size not stated in source
Bytes streamed per token	Nominal encoding width per precision	fp16, int8, 2-bit models	Measured via ANE memory-controller counters; hardware generation not stated in source

For the result to transfer to your case, the following must hold: your target SoC shares the ANE generation used in the study (not stated in source); your model uses operations covered by the 64-shape primitive matrix; and your CoreML version compiles the same graph structures. None of these are verified externally.

Results

Placement is expression-sensitive, not arithmetic-sensitive. A fused RMSNorm is fully ANE-eligible; its arithmetically identical decomposition routes entirely to CPU. The 64-shape matrix was built to isolate exactly this variable. The authors attribute the routing difference to the CoreML graph compiler responding to operation form rather than numerical equivalence.

The conv-heavy fp16 design at 25.85M parameters receives zero ANE bytes. Memory-controller counters confirm it. Quantizing the same graph to int8 or 2-bit returns ANE residency to approximately 83%, and under those conditions the decode speedup over the fp16 CPU-only baseline is 1.8–2.2×. The source does not state a minimum detectable effect size for the residency measurement. ~83% is a reported measurement, not a bound.

Architecture determines fp16 residency independently of quantization. The 22.29M all-attention fp16 model hits 98.9% ANE residency without quantization. The attribution goes to the all-attention architecture presenting a different operation mix to the compiler. No intermediate architecture is reported between 0% and 98.9%; the transition point is not established.

Decode cost follows a single bandwidth law across tested precisions. Bytes streamed per token land at approximately 0.77 of nominal encoding width, and that fraction is constant across fp16, int8, and 2-bit. No variance or confidence interval is stated in the source for this figure.

25M ternary models cluster tightly in memory and latency. Every 25M ternary model measured lands within 10.0–10.8 MB and 0.62–0.64 ms/token. The half-attention ternary at 25M: 10.5 MB, 0.63 ms/token. The 50M variant: 16.8 MB, 0.86 ms/token. Relative to the conv-heavy fp16 design the work began with, the 25M half-attention ternary is 9.8× smaller and 3.0× faster; the 50M is 6.1× smaller and 2.2× faster. These ratios are relative to one specific baseline. They describe nothing about speedups over other fp16 architectures.

Reported by 2608.22110 x· scroll →

Conditions around the comparison

scroll →

25.85M

conv-heavy fp16 model parameter count

22.29M

all-attention fp16 model parameter count

~83%

conv-heavy int8/2-bit residency

~0.77

bytes streamed per token fraction of nominal encoding width

Where results stop

Hardware generation. The specific Apple SoC is not stated in the source. ANE implementations differ across A-series and M-series generations; placement rules, compiler behaviour, and bandwidth characteristics may not transfer.

Scale. The largest model tested is 50M parameters. The bandwidth law and placement patterns are not established above that. Larger models simply weren’t in scope.

Task quality. This is a measurement study of placement and decode speed. Whether ternary or 2-bit quantization at these parameter counts preserves accuracy on downstream tasks is not addressed. The study holds “what is computed” fixed at the arithmetic level, which is distinct from task-level equivalence.

Precision coverage. The ~0.77 fraction is measured across fp16, int8, and 2-bit. 4-bit is not included. The source does not state whether this was underpowered to test or simply not run.

Primitive matrix coverage. Operations outside the 64-shape matrix were neither tested nor cleared. The source does not enumerate which operations are absent.

Null result note. No arm in the supplied results is reported as finding no effect. The conv-heavy fp16 model’s zero ANE residency is a finding, not a null. The study does not report any comparison that failed to detect a difference between conditions.

What to do with this

If you are compiling LLMs for CoreML, the primary practical point is that compiler placement annotations and actual ANE byte traffic are not the same thing. The memory-controller counter approach is the ground-truth method here; any profiling tool that reports only “intended” device may misrepresent what executed. A conv-heavy fp16 model routing entirely to CPU with no placement telemetry? Quantizing to int8 or 2-bit is the first lever. It recovered ~83% residency in the authors’ measurement.

The tight clustering of 25M ternary models (10.0–10.8 MB, 0.62–0.64 ms/token) suggests that at this parameter scale, weight encoding dominates architecture choice in reaching the ANE bandwidth floor. The ANEForge 2.28× speculative decode result reached a different optimisation axis on the same hardware class; the two measurement approaches are not directly comparable.

If your target is a different ANE generation, a parameter scale above 50M, or a precision not covered here, this study does not give you a number to use.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from a single source — the arXiv abstract for 2608.22110. The claims and quantities it worked from were extracted from that document by qwen/qwen3.8-27b. Before publication an automated gate re-checked every extracted claim against the source document (13 claim(s) and 17 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (0 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run 2608.22110’s benchmarks; figures attributed to it are its own.

FAQ

Does the 0.77 bandwidth fraction let me predict decode latency from model size?

The study reports ~0.77 as the ratio of bytes streamed per token to nominal encoding width, held constant across fp16, int8, and 2-bit. If you know nominal encoding width and ANE memory bandwidth for your SoC, the relationship holds as described — but the SoC bandwidth figure must come from your own measurements; the study does not supply it. The fraction applies only to the three precisions tested. 4-bit is not covered, and the source does not extend the claim there.

Why does the all-attention fp16 model reach 98.9% residency while the conv-heavy fp16 model reaches 0%?

The attribution goes to the operation mix each architecture presents to the CoreML compiler. Conv-heavy graphs include operations that, in fp16 form, route to CPU; all-attention graphs do not trigger the same routing. The 64-shape matrix and matched model comparisons establish the pattern, but the exact operation or graph threshold responsible for the switch is not identified.

Is ternary quantization distinct from 2-bit quantization in these measurements?

The study reports ternary models and 2-bit models as separate conditions. Ternary weights use three discrete values rather than arbitrary 2-bit integers, so the encoding and ANE mapping may differ even at the same nominal bit width. The tight ternary cluster (10.0–10.8 MB, 0.62–0.64 ms/token for all 25M ternary variants) is reported separately from the ~83% residency result for the conv-heavy graph in 2-bit. The source does not state whether those 2-bit models are also ternary.

How do I verify what is actually running on the ANE versus CPU?

ANE memory-controller byte counters read during inference. That's the method the authors describe as establishing what actually ran rather than what the compiler intended. The source states the method but does not specify the API or tooling path used to access those counters. Xcode Instruments' CoreML performance template is the obvious candidate, but the study does not name it and this note does not supply that inference.