# gpt-oss-20b scores range 0% to 87% by harness The same gpt-oss-20b weights produce scores from ~0% to ~87% depending on template, tool format, wire API, and reasoning effort. Canonical page: https://makerportal.ai/blog/itria-hackernews-49523381 Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-09-02 Section: Correction / Local LLMs · 5 min read Tags: local-llm --- The same gpt-oss-20b weights produce scores ranging from approximately 0% to approximately 87% depending on harness configuration — template, tool format, wire API endpoint, and reasoning effort level. Deployment quality belongs to the model-and-harness pair, not the model alone. The harness can shift effective capability by two orders of magnitude on its own. **Conditions attached to the figures** Burrito reports its evaluation ran gpt-oss-20b across 320,192 runs consuming 3.49B tokens over 1,062 GPU-hours on a single RTX 3090, at batch size 1, with 8 random seeds per configuration. Seven backend configurations were compared: burrito over llama.cpp and vLLM (with and without preserved thinking), vanilla llama.cpp with default and fixed jinja templates, and vanilla vLLM. Benchmarks covered the Big Function Calling Leaderboard (single-turn, live-API, multi-turn agentic), AIME25, and GPQA. **The cost of finding out your harness is broken** The arithmetic below bounds what this kind of discovery costs. $$ \text{Throughput} = \frac{T_{\text{total}}}{t_{\text{GPU}}} = \frac{3.49 \times 10^9 \text{ tokens}}{1{,}062 \times 3{,}600 \text{ s}} \approx 913 \text{ tok/s} $$ At batch size 1 on a single RTX 3090, that is the effective single-stream throughput for gpt-oss-20b across the full evaluation (prefill and decode mixed). Per-configuration cost: $$ C_{\text{config}} = \frac{1{,}062 \text{ GPU-h}}{7 \text{ configs} \times 8 \text{ seeds}} \approx 19.0 \text{ GPU-h per configuration-seed pair} $$ Each of the 56 configuration-seed pairs consumed roughly 62.3M tokens (3.49B / 56) and about 19 GPU-hours. The total discovery — that the default jinja template breaks the model, that the wire API choice multiplies your error rate, that structured schemas triple multi-turn accuracy — cost 1,062 hours on a single consumer card. The wire API finding carries the most operational risk. Burrito reports that on vLLM, the /v1/responses endpoint errors on 73.5–83.5% of multi-turn runs versus 29.0–36.0% on /v1/chat/completions. The ratio of error rates: $$ \frac{73.5}{36.0} \approx 2.0 \quad \text{to} \quad \frac{83.5}{29.0} \approx 2.9 $$ A client that selects /v1/responses over /v1/chat/completions on a vLLM backend degrades its effective reliability by a factor of 2–3 without changing a single model weight. The model is producing output that the /v1/responses parser cannot handle in multi-turn contexts at a rate roughly 2–3× higher than the /v1/chat/completions parser. Protocol-shape problem, not model problem. AIME25 introduces a separate axis. At a fixed ~1.4k reasoning-token budget, accuracy is 38% (low effort), 97% (medium), and 100% (high). The effort parameter swings 62 points without touching the model, template, or tool format. Native python tools add 21–24 points at low effort and 10–12 at medium. The interaction matters: at low effort, the model needs the python tool to compensate for insufficient reasoning; at high effort, the reasoning itself carries the load and the tool's marginal value shrinks. | Measurement | Condition | Value | |---|---|---| | Live-test accuracy | Default jinja template (commentary present) | ~3% | | Live-test accuracy | Commentary removed from valid output channels | ~40% | | Multi-turn error rate | vLLM, /v1/responses | 73.5–83.5% | | Multi-turn error rate | vLLM, /v1/chat/completions | 29.0–36.0% | | Multi-turn accuracy | Vanilla backends, AST parsing (fc_model=0) | ~0% | | AIME25 accuracy | ~1.4k reasoning tokens, low effort | 38% | | AIME25 accuracy | ~1.4k reasoning tokens, medium effort | 97% | | AIME25 accuracy | ~1.4k reasoning tokens, high effort | 100% | | Python tool gain | Low effort | +21–24 pts | | Python tool gain | Medium effort | +10–12 pts | No single knob accounts for the full 0%-to-87% range. The effects multiply: template correctness × wire API compatibility × tool schema format × effort level. **What this does not establish** The evaluation is single-model, single-GPU, batch-size-1. The findings are specific to gpt-oss-20b on an RTX 3090 with the particular llama.cpp and vLLM versions burrito tested. Transfer to other model families, other quantizations, or multi-GPU serving is not addressed. The 8 seeds per configuration give some statistical grounding, but the absolute accuracy figures at the low end (~3%, ~0%) sit in a range where a single lucky or unlucky seed could shift the point estimate meaningfully. **What this means for itria** gpt-oss-20b does not run on an iPhone or iPad — at 4-bit it needs roughly 12 GB against an on-device budget nearer 5 GB. What transfers is the harness lesson, not the model. Itria ships llama.cpp as its inference backend with Metal acceleration. Our work on [KV cache streaming on 16 GB CUDA](/blog/itria-hackernews-49511882) addresses the memory layer — keeping long contexts alive on constrained VRAM. Burrito's results suggest the harness layer above the backend is an orthogonal risk: the template that renders the conversation, the tool schema format, and the wire protocol the client speaks are all configuration surfaces that can independently zero out a model's effective capability. For a product that ships a fixed model on a fixed device, the harness is compiled into the app. It is not a runtime swappable dependency. Template and tool-format regressions are app regressions, and they should be caught in CI the same way a Metal shader compile failure would be. If you are sizing whether gpt-oss-20b fits on your target hardware before worrying about any of the above, the [VRAM and KV cache calculator](/lab/llm-vram-kvcache-calculator) will give you the memory ceiling.
Reported by iamskeole/burrito-core exact values· scroll →
320,192 runs
evaluation runs
3.49B tokens
tokens processed
1,062 GPU hours
GPU hours on RTX 3090
8 random seeds
number of configuration
Figure: exact values as iamskeole/burrito-core reports them, drawn from quantities extracted and quote-verified for this post. The cells do not share an axis and are not scaled against one another; the studio has not re-measured them.
--- *Method: this note was drafted by qwen/qwen3.8-27b from two sources — the published README of [iamskeole/burrito-core](https://github.com/iamskeole/burrito-core) and this studio's own published measurements, linked above. Before publication an automated gate re-checked every extracted claim against the source document (34 claim(s) and 18 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (7 derived from it, 19 row(s) supplied from our own tables). The studio has not re-run iamskeole/burrito-core's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Does this mean gpt-oss-20b is a bad model? No. The 100% AIME25 at high effort and the 87% upper bound show the model is capable. The 0% lower bound is a harness failure, not a model failure. The model is consistent; the harness around it is not. ### Can I reproduce these numbers on my hardware? The evaluation used a single RTX 3090 (24 GB VRAM), batch size 1, and the specific llama.cpp/vLLM versions burrito pinned. Your throughput will differ with different hardware, but the relative configuration effects (template, wire API, tool format) should transfer because they are protocol-level, not compute-level. ### Why does /v1/responses fail so much worse than /v1/chat/completions? Burrito does not state the root cause in the README. The error rates (73.5–83.5% vs 29.0–36.0%) suggest the /v1/responses wire format is less tolerant of the model's actual output shape in multi-turn contexts. Whether this is a vLLM parsing bug, a format mismatch, or a model output distribution issue is not specified. ### Is burrito something I should run in production? That is a deployment decision outside the scope of this note. The evaluation demonstrates that the harness layer is a real variable, not a theoretical one. Whether burrito's specific implementation is production-ready depends on your threat model, your tooling requirements, and your tolerance for the AGPL-3.0 SearXNG and Grafana components it bundles.