The cheapest AI model won the benchmark

When you've got twenty AI coding models in front of you, the mental shortcut is to pick the most expensive one. The intuition says it costs more because it's better, and that rule doesn't fail often enough for you to notice. We used it too, until we stopped trusting it.

We set up a benchmark. The same task for all twenty, on autopilot, each one isolated in an empty folder with its highest reasoning variant. The uncomfortable result: the model with the best score cost three cents, and there are models that cost twenty times more and scored lower.

How you measure something like this

With three tasks of increasing difficulty. All of them require building an engine from scratch in TypeScript, with no external libraries:

The prompt is the same for all of them and lives outside their folders, so no one reads it as extra context. The score from 0 to 10 isn't a human's impression: it combines a hidden test suite the model doesn't know about, a judge's assessment of the code and tests, autonomy (how many times intervention was needed) and robustness. The judge is Claude Sonnet, a model that doesn't compete, and it works on a copy outside the repo. The hidden suite isn't published precisely so it can keep being used in future runs.

The cheap one won, and it's no accident

In RRULE, the medium-difficulty task, all twenty finished. Here are the ones that came out on top:

Three models tied at 9.8. The three-cent one and the fifty-eight-cent one got the same score; the price difference was twenty-fold and the time difference nearly five-fold. In SQL, the hard task, DeepSeek V4.1 Flash came out first again with 173 cases out of 173 and $0.08. GPT-5.6 Luna came second again in both.

The conclusion isn't "cheap is better." It's that price doesn't measure what you think it measures. When you pay more for a model you're paying for tokens, throughput, latency, availability or brand, not correctness on a bounded task. On a well-specified problem, a cheap model with its reasoning variant maxed out can hold its own perfectly well.

Difficulty doesn't spread evenly: it wipes out half the field

The most useful data point in the benchmark isn't who wins, it's how many finish:

• RRULE: 20 out of 20.
• SQL: 4 out of 19.
• Regex: 2 out of 20.

With the easy task, the table doesn't discriminate: almost everyone passes and the differences are marginal. With the hard one, the pack breaks apart and you see who actually holds up the problem. A benchmark with a single gentle task is good for ranking models by score, not for knowing which one holds up when things get ugly.

And there's an honesty decision baked into how a model that doesn't finish gets scored. Two models were manually stopped for taking too long or spending too much: one had complete, compiling code but no tests or README, the other was at 95%. They're scored on what they'd done and given zero autonomy, because a model that doesn't finish in a usable time isn't a usable model even if its half-finished code is good.

What the table doesn't say

The highest reasoning variant guarantees nothing. One model spent the whole session thinking and cut off without writing almost anything. The problem wasn't the model: it was the harness. opencode caps each response at 32,000 output tokens, reasoning included, so we raised that cap so thinking wouldn't get cut off midway. It's a reminder that in a benchmark you're also measuring the tool you're measuring with.

Isolation is measured, not assumed. Each model runs with access outside its folder denied, and the result logs whether it tried. One tried and got blocked. Another managed to read paths where other results were sitting, so its row is flagged as contaminated and doesn't count the same as the rest. Without that column you'd be comparing solutions that saw the answers, and you wouldn't even know it.

The judge has nerves. It's an LLM scoring others, with a variability of half a point up or down. It's not a lab measurement and we don't dress it up as one: when it re-runs, it reuses the stored verdict and only re-runs the hidden suite.

The efficiency frontier

The table we actually use isn't "who scores highest," it's score versus cost, and also versus tokens, steps and time. In the interactive version of the benchmark the score weights are adjustable, because prioritizing correctness isn't the same as prioritizing autonomy or robustness. The chart draws a frontier: above it, no model improves without spending more.

For a small studio, the question was never "which is the best model," but "which gives the most bang for the buck on this kind of task." A 9.8 at three cents beats a 9.8 at fifty-eight cents, and both beat a 6.3 at eighty-four cents that didn't even finish.

What we're taking away

• Price isn't a proxy for quality. It measures tokens, speed and availability. To pick a model you have to measure the other stuff, and measuring it isn't expensive.
• What discriminates is task difficulty. With an easy test everyone passes; the information is in the one that breaks the pack.
• The highest reasoning variant is no guarantee, and sometimes the bottleneck is your own harness.
• You have to measure isolation. A benchmark without contamination control compares solutions you don't think it's comparing.

If you're choosing an AI model for your team and don't know where to start, get in touch. The benchmark is public at github.com/rldona/opencode-models-benchmarks, with an interactive site to change the score weights and see the efficiency frontier.