# OpenMed's 24–33× MLX speedup is compute, not memory OpenMed's 24–33× MLX speedup is a compute advantage, not a memory one. Batch throughput (3.3× CPU, 2.2× MLX) shows both backends are partially… Canonical page: https://makerportal.ai/blog/itria-github-1069760430 Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-09-09 Section: Correction / On-Device AI · 5 min read Tags: itria, on-device-ai, ios-craft, privacy-arch --- The 24–33× MLX-over-CPU speedup OpenMed reports is a single-forward-pass (token classification) result. The accompanying batch throughput figures (3.3× on CPU, 2.2× on MLX) reveal that both backends remain partially bandwidth-bound for single-document inference. That is a compute advantage, not a memory one, and it does not transfer to the autoregressive decode regime where our [itria](/blog/2026-08-11-itria-hackernews-49245161) field note operates. ## What OpenMed is OpenMed is an Apache-2.0 SDK for clinical NER and PII de-identification, running entirely on-device across Python, Swift/MLX, Android/ONNX, and browser/Transformers.js runtimes. It targets healthcare data pipelines that need PII stripped before data leaves a local machine. The largest models in its manifest are 434M-parameter encoder-based token classifiers; the smallest are 109M. ## Figures and their conditions OpenMed's README states the following, each tied to a specific measurement context: - "MLX on Apple Silicon: 24–33× faster than CPU PyTorch for the Privacy Filter: median latency per inference step, lower is better." The unit is one forward pass over a document (or set of documents), not tokens generated. - "Batch processing: up to 3.3× higher throughput on CPU and 2.2× on MLX vs. one document at a time." The batch size is not stated. - Model parameter counts: 434M (disease, pharma, pii_superclinical_large) and 109M (anatomy, gene). Quantization level is not stated for the speedup claim. - 340M+ downloads, 10M+ installs (banner text; methodology not stated).

Reported by maziyarpanahi/openmed M· scroll →

Conditions around the comparison

scroll →

33
model-backed PII languages: 24–33×
faster: 3.3×
higher throughput on CPU: 109M
parameters for gene_detection_genecorpus

Figure: values as maziyarpanahi/openmed reports them, drawn from quantities extracted and quote-verified for this post. Bars are proportional within their shared unit; condition cells are exact values and are not scaled against one another. The studio has not re-measured them.

## The batch numbers expose the regime Batching $B$ documents relative to single-document processing gives a throughput speedup governed by: $$ S(B) \;=\; \frac{B \cdot t(1)}{t(B)} \;=\; \frac{B \cdot \max\!\left(\dfrac{W}{\mathrm{BW}},\;\dfrac{\mathcal{F}}{\mathrm{FLOP}}\right)}{\max\!\left(\dfrac{W}{\mathrm{BW}},\;\dfrac{B\,\mathcal{F}}{\mathrm{FLOP}}\right)} $$ where $W$ is total weight bytes, $\mathrm{BW}$ is memory bandwidth, $\mathcal{F}$ is FLOPs per document, and $\mathrm{FLOP}$ is peak compute throughput. For a single document ($B=1$), the pass is bandwidth-bound if $W/\mathrm{BW} > \mathcal{F}/\mathrm{FLOP}$, else compute-bound. For a batch, the weight read is amortized but compute scales linearly with $B$. Define $\alpha = \frac{W/\mathrm{BW}}{\mathcal{F}/\mathrm{FLOP}}$ — the ratio of bandwidth time to compute time for one document. Then: - If $B \le \alpha$: $t(B) = W/\mathrm{BW}$ (weights still dominate), so $S(B) = B$. - If $B > \alpha$: $t(B) = B\mathcal{F}/\mathrm{FLOP}$ (compute now dominates), so $S(B) = \alpha$. $S(B) \le B$ always. The observed CPU speedup of 3.3× implies $B \ge 4$ (smallest integer $\ge 3.3$). At $B = 4 > \alpha_{\text{CPU}}$, the CPU has entered the compute-saturated plateau, giving $\alpha_{\text{CPU}} \approx 3.3$. Same logic for MLX: $B \ge 3$ (smallest integer $\ge 2.2$), plateau gives $\alpha_{\text{MLX}} \approx 2.2$. The ratio of compute-to-bandwidth advantage between the two backends: $$ \frac{\alpha_{\text{CPU}}}{\alpha_{\text{MLX}}} = \frac{3.3}{2.2} = 1.5 $$ The MLX GPU's FLOP-per-byte-of-bandwidth is 1.5× the CPU's. On Apple Silicon, CPU and GPU share the same unified memory and the same physical bandwidth. The GPU wins because it has more arithmetic units per byte it can pull. | Measurement | Regime | Model size | Backend | Key figure | Source | |---|---|---|---|---|---| | OpenMed Privacy Filter | Single forward pass (token classification) | 434M / 109M (quant not stated) | MLX vs CPU PyTorch, Apple Silicon | 24–33× speedup | OpenMed README | | OpenMed batch, CPU | Batched forward pass | not stated | CPU PyTorch | 3.3× throughput vs single-doc | OpenMed README | | OpenMed batch, MLX | Batched forward pass | not stated | MLX | 2.2× throughput vs single-doc | OpenMed README | | itria 3B Q4, M4 Max | Autoregressive decode | 3B Q4 (1.8 GB GGUF) | llama.cpp / Metal, 400 GB/s | 220 tok/s | [our bench](/blog/2026-08-11-itria-hackernews-49245161) | | itria 7B Q4, Pi 5 | Autoregressive decode | 7B Q4 (4.2 GB GGUF) | llama.cpp, 17 GB/s | 4.0 tok/s | [our bench](/blog/2026-08-11-itria-hackernews-49245161) | ## What cannot be concluded The batch size $B$ is not stated in the README. The derivation above assumes $B$ is small enough (4 or 5) that the observed speedup sits on the compute-saturated plateau. If $B$ were larger (say 8 or 16), the CPU could still be in the linear-scaling region ($S = B$ would require $B = 3.3$, which is not an integer, so this is already ruled out for the CPU). For MLX, $S = 2.2$ with $B = 3$ is consistent, but so is $B = 5$ with $\alpha_{\text{MLX}} = 2.2$. The exact batch size and quantization level are not recoverable from the supplied figures. ## What this means for a deployment decision If you are evaluating OpenMed for a PII-stripping pipeline on Apple Silicon hardware, the 24–33× number is real and relevant — but it tells you the MLX path will be fast for the *classification step*, not that the end-to-end pipeline will be 24–33× faster than a CPU pipeline. The CPU path is only 3.3× slower under batching, and for a single document the gap is narrower still (both are partially bandwidth-bound on shared memory). For itria specifically: encoder forward passes and autoregressive decodes live in different regimes. A 434M classifier on M4 Max will be fast regardless of backend — the weight footprint (0.87 GB in fp16) is small relative to the 400 GB/s bandwidth, and the pass is a single shot. A 7B decoder on the same chip is bandwidth-bound every token. Switching backends does not change that. The [PrivateRedact comparison](/blog/2026-08-11-itria-hackernews-49245161) showed a 7B Q4 model collapsing to 4.0 tok/s on a Pi 5 because 4.2 GB of weights must be read from 17 GB/s of bandwidth per token. No batching trick fixes that; it is a hard ceiling. OpenMed's models sidestep the problem entirely by not generating tokens. --- *Method: this note was drafted by qwen/qwen3.8-27b from two sources — the published README of [maziyarpanahi/openmed](https://github.com/maziyarpanahi/openmed) and this studio's own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (18 claim(s) and 17 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (3 derived from it, 18 row(s) supplied from our own tables). The studio has not re-run maziyarpanahi/openmed's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Does the 24–33× speedup apply to the 109M models too? The README attaches the speedup claim to "the Privacy Filter" without specifying which model in the manifest. The 109M and 434M models share the same architecture family (encoder token classifiers), so the regime is the same, but the absolute latency will differ. The range (24 to 33) likely spans different model sizes or sequence lengths, but the README does not break it down. ### Can I use the batch throughput number to estimate latency for my document volume? Not directly. The 3.3× and 2.2× figures are relative to single-document throughput on unspecified hardware and with an unstated batch size. You would need the absolute single-document latency on your target hardware to compute wall-clock time for a batch. ### Why is the MLX batch speedup (2.2×) lower than the CPU batch speedup (3.3×)? Because the GPU is more compute-saturated. Batching helps most when the bottleneck is reading weights from memory (you read them once for the whole batch). The GPU's high FLOP rate means compute, not bandwidth, is the limiting factor sooner, so batching amortizes less. The CPU, with fewer ALUs, stays bandwidth-bound longer and benefits more from amortizing the weight read. ### Does the 340M+ download figure tell me anything about reliability? No. The figure comes from the README banner image alt-text. No methodology, time window, or package registry is cited. It is not a measurement the studio has verified.