M3 Ultra decode: 9.8 to 139 tok/s measured

Every model card states parameters and quantisation. Almost none states what the thing decodes at on hardware you can buy. Across 9 resident models the decode rate spans 9.8–139 tok/s, a 14.3x range — openai/gpt-oss-20b fastest, nousresearch/hermes-4-70b slowest.

What was measured

9 models already resident in LM Studio on Apple M3 Ultra, 256 GB, macOS 26.5.2. Three quantities per model, each the median of 3 runs:

• Decode — generated tokens per second over a run of 256 tokens at temperature 0.
• Prefill — prompt tokens divided by time to first token on a long prompt. How fast it reads.
• First token — time to first token on a short prompt. Interactive latency.

Measured by MakerPortal Studio tok/s· scroll →

What the numbers say

• Same weights, 3 ways: qwen2.5-coder-32b-instruct decodes at 35.3 tok/s (MLX 4bit), 20.1 tok/s (MLX 8bit), 19.3 tok/s (llama.cpp Q8_0). That is a 1.83x span with the model held fixed — the only comparison here where nothing else varies.
• And on the same weights prefill barely moves: 1.12x against decode’s 1.83x (362.7 tok/s vs 359.9 tok/s vs 323.3 tok/s). The usual reading is that decode is bandwidth-bound — fewer bits per weight means fewer bytes moved per token — while prefill is compute-bound and processes the prompt in parallel regardless. This protocol measures neither bandwidth nor FLOPs, so that is the explanation and not the finding; the finding is that one number moved 1.6x more than the other on the same model.
• llama.cpp medians 41.6 tok/s across its 4 models against 23.5 tok/s for MLX’s 5 — a 1.77x gap. The two sets are different models, so this is a gap between what is available on each runtime here, not a controlled comparison.
• Prefill outruns decode by 10.3x to 22.4x across the 9 models where both were measured (widest: qwen/qwen3.5-9b). A long prompt costs far less than the answer to it.

Every row, with its conditions

Model	Backend	Quant	Decode	Prefill	First token	Conditions
openai/gpt-oss-20b	MLX	MXFP4	139 tok/s	2526.3 tok/s	0.19 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; mlx-llm-mac-arm64-apple-metal-advsimd 1.11.0; median of 3 runs of 255 tokens, spread 1.01x
openai/gpt-oss-120b	llama.cpp	MXFP4	87.4 tok/s	1115 tok/s	0.24 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; llama.cpp-mac-arm64-apple-metal-advsimd 2.29.1; median of 3 runs of 256 tokens, spread 1x
qwen/qwen3.5-9b	llama.cpp	Q8_0	63.9 tok/s	1430.2 tok/s	0.11 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; llama.cpp-mac-arm64-apple-metal-advsimd 2.29.1; median of 3 runs of 256 tokens, spread 1x
qwen2.5-coder-32b-instruct-mlx@4bit	MLX	4bit	35.3 tok/s	362.7 tok/s	0.43 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; mlx-llm-mac-arm64-apple-metal-advsimd 1.11.0; median of 3 runs of 255 tokens, spread 1x
qwen/qwen3.8-27b	MLX	8bit	23.5 tok/s	421.6 tok/s	0.42 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; mlx-llm-mac-arm64-apple-metal-advsimd 1.11.0; median of 3 runs of 255 tokens, spread 1x
google/gemma-4-31b	MLX	8bit	20.1 tok/s	330.4 tok/s	0.39 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; mlx-llm-mac-arm64-apple-metal-advsimd 1.11.0; median of 3 runs of 255 tokens, spread 1.02x
qwen2.5-coder-32b-instruct-mlx@8bit	MLX	8bit	20.1 tok/s	359.9 tok/s	0.47 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; mlx-llm-mac-arm64-apple-metal-advsimd 1.11.0; median of 3 runs of 255 tokens, spread 1.01x
qwen2.5-coder-32b-instruct	llama.cpp	Q8_0	19.3 tok/s	323.3 tok/s	0.33 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; llama.cpp-mac-arm64-apple-metal-advsimd 2.29.1; median of 3 runs of 256 tokens, spread 1x
nousresearch/hermes-4-70b	llama.cpp	Q8_0	9.8 tok/s	159.5 tok/s	0.64 s	Apple M3 Ultra, 256 GB, macOS 26.5.2; llama.cpp-mac-arm64-apple-metal-advsimd 2.29.1; median of 3 runs of 256 tokens, spread 1x

What this does not show

Nothing about quality. A model that decodes twice as fast and answers worse is worse, and this protocol never reads a single output. Nothing about batching or concurrency either — every run is one request at a time, which is the desktop case and not the serving case. And only qwen2.5-coder-32b-instruct is a controlled comparison; every other pair on this page is two different models, and the gap between them describes what is installed here rather than the runtimes themselves.

Method

Protocol mp-decode-v3. One warm-up call per model, discarded, so no measurement includes model load or Metal pipeline compilation. Then 3 decode runs and 3 prefill runs, each with a per-run prompt prefix so the inference server’s prompt cache cannot restore a previous run — the prefix is seeded by the run index rather than a clock, so re-running the protocol sends identical bytes. Rates come from LM Studio’s own /api/v0 statistics; the prefill rate is computed as prompt tokens over time to first token, because no endpoint reports it. Runs that stopped early are dropped and the count of drops is carried in the row. The harness is scripts/trends/local-bench.mjs; its protocol constants are versioned so two benches taken months apart are never silently mixed.

FAQ

Why is the prompt different on every run?

LM Studio caches prompt prefixes and reports doing so in its own log. Sending the same benchmark prompt three times means runs two and three measure the cache rather than the model, which shows up as a time-to-first-token that improves the more you measure it. Every prompt here opens with a per-run prefix derived from the run index, so the shared prefix is zero. It is derived from the index and not from a clock, so re-running the protocol sends identical bytes and two benches stay comparable.

Do these numbers transfer to other hardware?

No. Every figure here is bound to Apple M3 Ultra, 256 GB running macOS 26.5.2, at the quantisation and runtime version named in its own row. Decode on Apple silicon is largely a memory-bandwidth story, so a machine with different bandwidth will not scale these numbers by a constant.

Is a faster backend the same as a better one?

Partly. Most rows here compare different models at different quantisations, so that gap is between what is installed on each side rather than between the runtimes. But qwen2.5-coder-32b-instruct is present 3 ways — same weights, same machine, same protocol — and that one IS controlled: it varies the serving and nothing else. Read the rest as a description of this machine.

What was dropped, and why?

A decode run shorter than 100 tokens is discarded, because a rate over a very short window is dominated by sampling noise — the same model measured in the same minute gave 37.5 tok/s over four tokens and 28.1 over eight, then 35.6 three times running over 127. The floor is absolute rather than a fraction of the 256 tokens requested: a model that answers the question completely and stops is a measurement, not a failure, and an earlier version of this harness refused exactly those rows. Any model whose runs were all below the floor is reported as refused rather than as fast, and the median length each rate was measured over travels with its row.