OpenMed's 24–33× MLX speedup is compute, not memory

The 24–33× MLX-over-CPU speedup OpenMed reports is a single-forward-pass (token classification) result. The accompanying batch throughput figures (3.3× on CPU, 2.2× on MLX) reveal that both backends remain partially bandwidth-bound for single-document inference. That is a compute advantage, not a memory one, and it does not transfer to the autoregressive decode regime where our itria field note operates.

What OpenMed is

OpenMed is an Apache-2.0 SDK for clinical NER and PII de-identification, running entirely on-device across Python, Swift/MLX, Android/ONNX, and browser/Transformers.js runtimes. It targets healthcare data pipelines that need PII stripped before data leaves a local machine. The largest models in its manifest are 434M-parameter encoder-based token classifiers; the smallest are 109M.

Figures and their conditions

OpenMed’s README states the following, each tied to a specific measurement context:

• “MLX on Apple Silicon: 24–33× faster than CPU PyTorch for the Privacy Filter: median latency per inference step, lower is better.” The unit is one forward pass over a document (or set of documents), not tokens generated.
• “Batch processing: up to 3.3× higher throughput on CPU and 2.2× on MLX vs. one document at a time.” The batch size is not stated.
• Model parameter counts: 434M (disease, pharma, pii_superclinical_large) and 109M (anatomy, gene). Quantization level is not stated for the speedup claim.
• 340M+ downloads, 10M+ installs (banner text; methodology not stated).

Reported by maziyarpanahi/openmed M· scroll →

Conditions around the comparison

scroll →

33

model-backed PII languages

24–33×

faster

3.3×

higher throughput on CPU

109M

parameters for gene_detection_genecorpus

The batch numbers expose the regime

Batching $B$ documents relative to single-document processing gives a throughput speedup governed by:

$$S(B) \;=\; \frac{B \cdot t(1)}{t(B)} \;=\; \frac{B \cdot \max\!\left(\dfrac{W}{\mathrm{BW}},\;\dfrac{\mathcal{F}}{\mathrm{FLOP}}\right)}{\max\!\left(\dfrac{W}{\mathrm{BW}},\;\dfrac{B\,\mathcal{F}}{\mathrm{FLOP}}\right)}$$

where $W$ is total weight bytes, $\mathrm{BW}$ is memory bandwidth, $\mathcal{F}$ is FLOPs per document, and $\mathrm{FLOP}$ is peak compute throughput. For a single document ($B=1$), the pass is bandwidth-bound if $W/\mathrm{BW} > \mathcal{F}/\mathrm{FLOP}$, else compute-bound. For a batch, the weight read is amortized but compute scales linearly with $B$.

Define $\alpha = \frac{W/\mathrm{BW}}{\mathcal{F}/\mathrm{FLOP}}$ — the ratio of bandwidth time to compute time for one document. Then:

• If $B \le \alpha$: $t(B) = W/\mathrm{BW}$ (weights still dominate), so $S(B) = B$.
• If $B > \alpha$: $t(B) = B\mathcal{F}/\mathrm{FLOP}$ (compute now dominates), so $S(B) = \alpha$.

$S(B) \le B$ always. The observed CPU speedup of 3.3× implies $B \ge 4$ (smallest integer $\ge 3.3$). At $B = 4 > \alpha_{\text{CPU}}$, the CPU has entered the compute-saturated plateau, giving $\alpha_{\text{CPU}} \approx 3.3$. Same logic for MLX: $B \ge 3$ (smallest integer $\ge 2.2$), plateau gives $\alpha_{\text{MLX}} \approx 2.2$.

The ratio of compute-to-bandwidth advantage between the two backends:

$$\frac{\alpha_{\text{CPU}}}{\alpha_{\text{MLX}}} = \frac{3.3}{2.2} = 1.5$$

The MLX GPU’s FLOP-per-byte-of-bandwidth is 1.5× the CPU’s. On Apple Silicon, CPU and GPU share the same unified memory and the same physical bandwidth. The GPU wins because it has more arithmetic units per byte it can pull.

Measurement	Regime	Model size	Backend	Key figure	Source
OpenMed Privacy Filter	Single forward pass (token classification)	434M / 109M (quant not stated)	MLX vs CPU PyTorch, Apple Silicon	24–33× speedup	OpenMed README
OpenMed batch, CPU	Batched forward pass	not stated	CPU PyTorch	3.3× throughput vs single-doc	OpenMed README
OpenMed batch, MLX	Batched forward pass	not stated	MLX	2.2× throughput vs single-doc	OpenMed README
itria 3B Q4, M4 Max	Autoregressive decode	3B Q4 (1.8 GB GGUF)	llama.cpp / Metal, 400 GB/s	220 tok/s	our bench
itria 7B Q4, Pi 5	Autoregressive decode	7B Q4 (4.2 GB GGUF)	llama.cpp, 17 GB/s	4.0 tok/s	our bench

What cannot be concluded

The batch size $B$ is not stated in the README. The derivation above assumes $B$ is small enough (4 or 5) that the observed speedup sits on the compute-saturated plateau. If $B$ were larger (say 8 or 16), the CPU could still be in the linear-scaling region ($S = B$ would require $B = 3.3$, which is not an integer, so this is already ruled out for the CPU). For MLX, $S = 2.2$ with $B = 3$ is consistent, but so is $B = 5$ with $\alpha_{\text{MLX}} = 2.2$. The exact batch size and quantization level are not recoverable from the supplied figures.

What this means for a deployment decision

If you are evaluating OpenMed for a PII-stripping pipeline on Apple Silicon hardware, the 24–33× number is real and relevant — but it tells you the MLX path will be fast for the classification step, not that the end-to-end pipeline will be 24–33× faster than a CPU pipeline. The CPU path is only 3.3× slower under batching, and for a single document the gap is narrower still (both are partially bandwidth-bound on shared memory).

For itria specifically: encoder forward passes and autoregressive decodes live in different regimes. A 434M classifier on M4 Max will be fast regardless of backend — the weight footprint (0.87 GB in fp16) is small relative to the 400 GB/s bandwidth, and the pass is a single shot. A 7B decoder on the same chip is bandwidth-bound every token. Switching backends does not change that. The PrivateRedact comparison showed a 7B Q4 model collapsing to 4.0 tok/s on a Pi 5 because 4.2 GB of weights must be read from 17 GB/s of bandwidth per token. No batching trick fixes that; it is a hard ceiling. OpenMed’s models sidestep the problem entirely by not generating tokens.

---

Method: this note was drafted by qwen/qwen3.8-27b from two sources — the published README of maziyarpanahi/openmed and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (18 claim(s) and 17 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (3 derived from it, 18 row(s) supplied from our own tables). The studio has not re-run maziyarpanahi/openmed’s benchmarks; figures attributed to it are its own.

FAQ

Does the 24–33× speedup apply to the 109M models too?

The README attaches the speedup claim to "the Privacy Filter" without specifying which model in the manifest. The 109M and 434M models share the same architecture family (encoder token classifiers), so the regime is the same, but the absolute latency will differ. The range (24 to 33) likely spans different model sizes or sequence lengths, but the README does not break it down.

Can I use the batch throughput number to estimate latency for my document volume?

Not directly. The 3.3× and 2.2× figures are relative to single-document throughput on unspecified hardware and with an unstated batch size. You would need the absolute single-document latency on your target hardware to compute wall-clock time for a batch.

Why is the MLX batch speedup (2.2×) lower than the CPU batch speedup (3.3×)?

Because the GPU is more compute-saturated. Batching helps most when the bottleneck is reading weights from memory (you read them once for the whole batch). The GPU's high FLOP rate means compute, not bandwidth, is the limiting factor sooner, so batching amortizes less. The CPU, with fewer ALUs, stays bandwidth-bound longer and benefits more from amortizing the weight read.

Does the 340M+ download figure tell me anything about reliability?

No. The figure comes from the README banner image alt-text. No methodology, time window, or package registry is cited. It is not a measurement the studio has verified.