gpt-oss-20b scores range 0% to 87% by harness

The same gpt-oss-20b weights produce scores ranging from approximately 0% to approximately 87% depending on harness configuration — template, tool format, wire API endpoint, and reasoning effort level. Deployment quality belongs to the model-and-harness pair, not the model alone. The harness can shift effective capability by two orders of magnitude on its own.

Conditions attached to the figures

Burrito reports its evaluation ran gpt-oss-20b across 320,192 runs consuming 3.49B tokens over 1,062 GPU-hours on a single RTX 3090, at batch size 1, with 8 random seeds per configuration. Seven backend configurations were compared: burrito over llama.cpp and vLLM (with and without preserved thinking), vanilla llama.cpp with default and fixed jinja templates, and vanilla vLLM. Benchmarks covered the Big Function Calling Leaderboard (single-turn, live-API, multi-turn agentic), AIME25, and GPQA.

The cost of finding out your harness is broken

The arithmetic below bounds what this kind of discovery costs.

$$\text{Throughput} = \frac{T_{\text{total}}}{t_{\text{GPU}}} = \frac{3.49 \times 10^9 \text{ tokens}}{1{,}062 \times 3{,}600 \text{ s}} \approx 913 \text{ tok/s}$$

At batch size 1 on a single RTX 3090, that is the effective single-stream throughput for gpt-oss-20b across the full evaluation (prefill and decode mixed). Per-configuration cost:

$$C_{\text{config}} = \frac{1{,}062 \text{ GPU-h}}{7 \text{ configs} \times 8 \text{ seeds}} \approx 19.0 \text{ GPU-h per configuration-seed pair}$$

Each of the 56 configuration-seed pairs consumed roughly 62.3M tokens (3.49B / 56) and about 19 GPU-hours. The total discovery — that the default jinja template breaks the model, that the wire API choice multiplies your error rate, that structured schemas triple multi-turn accuracy — cost 1,062 hours on a single consumer card.

The wire API finding carries the most operational risk. Burrito reports that on vLLM, the /v1/responses endpoint errors on 73.5–83.5% of multi-turn runs versus 29.0–36.0% on /v1/chat/completions. The ratio of error rates:

$$\frac{73.5}{36.0} \approx 2.0 \quad \text{to} \quad \frac{83.5}{29.0} \approx 2.9$$

A client that selects /v1/responses over /v1/chat/completions on a vLLM backend degrades its effective reliability by a factor of 2–3 without changing a single model weight. The model is producing output that the /v1/responses parser cannot handle in multi-turn contexts at a rate roughly 2–3× higher than the /v1/chat/completions parser. Protocol-shape problem, not model problem.

AIME25 introduces a separate axis. At a fixed ~1.4k reasoning-token budget, accuracy is 38% (low effort), 97% (medium), and 100% (high). The effort parameter swings 62 points without touching the model, template, or tool format. Native python tools add 21–24 points at low effort and 10–12 at medium. The interaction matters: at low effort, the model needs the python tool to compensate for insufficient reasoning; at high effort, the reasoning itself carries the load and the tool’s marginal value shrinks.

Measurement	Condition	Value
Live-test accuracy	Default jinja template (commentary present)	~3%
Live-test accuracy	Commentary removed from valid output channels	~40%
Multi-turn error rate	vLLM, /v1/responses	73.5–83.5%
Multi-turn error rate	vLLM, /v1/chat/completions	29.0–36.0%
Multi-turn accuracy	Vanilla backends, AST parsing (fc_model=0)	~0%
AIME25 accuracy	~1.4k reasoning tokens, low effort	38%
AIME25 accuracy	~1.4k reasoning tokens, medium effort	97%
AIME25 accuracy	~1.4k reasoning tokens, high effort	100%
Python tool gain	Low effort	+21–24 pts
Python tool gain	Medium effort	+10–12 pts

No single knob accounts for the full 0%-to-87% range. The effects multiply: template correctness × wire API compatibility × tool schema format × effort level.

What this does not establish

The evaluation is single-model, single-GPU, batch-size-1. The findings are specific to gpt-oss-20b on an RTX 3090 with the particular llama.cpp and vLLM versions burrito tested. Transfer to other model families, other quantizations, or multi-GPU serving is not addressed. The 8 seeds per configuration give some statistical grounding, but the absolute accuracy figures at the low end (~3%, ~0%) sit in a range where a single lucky or unlucky seed could shift the point estimate meaningfully.

What this means for itria

gpt-oss-20b does not run on an iPhone or iPad — at 4-bit it needs roughly 12 GB against an on-device budget nearer 5 GB. What transfers is the harness lesson, not the model.

Itria ships llama.cpp as its inference backend with Metal acceleration. Our work on KV cache streaming on 16 GB CUDA addresses the memory layer — keeping long contexts alive on constrained VRAM. Burrito’s results suggest the harness layer above the backend is an orthogonal risk: the template that renders the conversation, the tool schema format, and the wire protocol the client speaks are all configuration surfaces that can independently zero out a model’s effective capability. For a product that ships a fixed model on a fixed device, the harness is compiled into the app. It is not a runtime swappable dependency. Template and tool-format regressions are app regressions, and they should be caught in CI the same way a Metal shader compile failure would be.

If you are sizing whether gpt-oss-20b fits on your target hardware before worrying about any of the above, the VRAM and KV cache calculator will give you the memory ceiling.

Reported by iamskeole/burrito-core exact values· scroll →

320,192 runs

evaluation runs

3.49B tokens

tokens processed

1,062 GPU hours

GPU hours on RTX 3090

8 random seeds

number of configuration

---

Method: this note was drafted by qwen/qwen3.8-27b from two sources — the published README of iamskeole/burrito-core and this studio’s own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (34 claim(s) and 18 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (7 derived from it, 19 row(s) supplied from our own tables). The studio has not re-run iamskeole/burrito-core’s benchmarks; figures attributed to it are its own.

FAQ

Does this mean gpt-oss-20b is a bad model?

No. The 100% AIME25 at high effort and the 87% upper bound show the model is capable. The 0% lower bound is a harness failure, not a model failure. The model is consistent; the harness around it is not.

Can I reproduce these numbers on my hardware?

The evaluation used a single RTX 3090 (24 GB VRAM), batch size 1, and the specific llama.cpp/vLLM versions burrito pinned. Your throughput will differ with different hardware, but the relative configuration effects (template, wire API, tool format) should transfer because they are protocol-level, not compute-level.

Why does /v1/responses fail so much worse than /v1/chat/completions?

Burrito does not state the root cause in the README. The error rates (73.5–83.5% vs 29.0–36.0%) suggest the /v1/responses wire format is less tolerant of the model's actual output shape in multi-turn contexts. Whether this is a vLLM parsing bug, a format mismatch, or a model output distribution issue is not specified.

Is burrito something I should run in production?

That is a deployment decision outside the scope of this note. The evaluation demonstrates that the harness layer is a real variable, not a theoretical one. Whether burrito's specific implementation is production-ready depends on your threat model, your tooling requirements, and your tolerance for the AGPL-3.0 SearXNG and Grafana components it bundles.