AI Coding Benchmarks: Why Cost Per Task Beats Leaderboards

The median engineering team sees just a 7.76% PR throughput improvement from AI coding tools, despite vendor promises of 3x productivity. That’s not a rounding error — it’s the gap between marketing and measured reality across 400+ organizations tracked over 14 months by DX. Most teams land in the 5–15% range. Meaningful, but nowhere near the order-of-magnitude gains being sold.

Here’s where it gets interesting. The models competing for your coding budget have converged in capability — the top tier is crowded, and open-source options are in it. But their task costs vary by up to 12x. The cheapest model per token and the lowest-ranked agent on a leaderboard each outperformed pricier leaders on cost per working output in recent controlled tests. Price and benchmark rank, it turns out, inversely predict delivered value.

If you’re evaluating AI coding tools by staring at leaderboard screenshots, you’re making your most expensive decision on the least reliable data. The benchmarks that actually matter aren’t the ones vendors put on their slides — they’re the ones you run on your own codebase, measuring dollars per shipped fix.

The Leaderboard Illusion: Why Benchmark Rank Misleads

GPT-5.6 Sol tops the Artificial Analysis Coding Agent Index at a score of 80, about 2.8 points ahead of Claude Fable 5 at 77.2, per Capital and Compute. That sounds decisive. It isn’t.

Flip to SWE-Bench Pro and the ranking inverts: Claude Fable 5 scores 80.3% against Sol’s 64.6% on the same benchmark. On Google’s Android Bench with the Harbor framework, Fable 5 leads at 84.5%, followed by GPT-5.5 at 80.2% and Claude Sonnet 5 at 76.2%. Different benchmark, different winner, different procurement decision.

The problem runs deeper than which test you pick. Benchmark contamination — where models partly recall solutions from training data — is documented and widespread. Cursor itself admitted that Grok 4.5 had an advantage on CursorBench because “an earlier snapshot of the Cursor codebase was accidentally included in training.” OpenAI abandoned SWE-bench Verified after finding roughly 60% of a sampled problem set had material flaws. Cognition’s FrontierCode 1.1 is Cognition’s own benchmark, scored by Cognition, promoting Cognition’s model.

Every lab grades its own homework. Not by inventing private tests, but by choosing which public score to put on the slide and which run to report. What outsiders rarely see is how many attempts a result took, which tasks were quietly dropped, and how the rubric was weighted. The harness matters more than the model — agent scaffolding can shift scores dramatically without changing the underlying model. Real-world coding agent performance runs roughly half of reported leaderboard scores.

The 12x Price Gap Nobody Talks About

Output token prices for frontier coding models span a 12x gap from $4.25 per million tokens (Meta Muse Spark 1.1) to $50 per million (Claude Fable 5), per Beri. That’s not a marginal difference. It’s the difference between a sustainable CI pipeline and one you disable on Fridays.

Here’s the thing: per-token pricing is nearly useless for agentic workloads. What matters is cost per completed task — and that metric tells a completely different story. Artificial Analysis estimated Grok 4.5 at $2.49 per coding task, versus $5.07 for GPT-5.5 in Codex and $11.80 for Fable 5 in Claude Code. GPT-5.6 Sol’s cost per intelligence task is $1.04, about a third of Fable 5’s cost, per Capital and Compute.

The cheapest model isn’t winning because it’s cutting corners on quality. It’s winning because mixture-of-experts architectures route each token to a small subset of specialized subnetworks rather than activating the entire parameter set. Grok 4.5 resolves coding tasks using an average of 15,954 output tokens against Opus 4.8’s 67,020 — a 4.2x efficiency advantage that flows from architecture, not from underinvestment in capability.

Model	Output $/M Tokens	Cost Per Task	Key Benchmark	Source
Meta Muse Spark 1.1	$4.25	~$0.26	AA Intelligence Index: 71	TechTimes
Grok 4.5	—	$2.49	Terminal-Bench 2.1: 83.3%	Computerworld
GPT-5.6 Sol	—	$1.04	AA Coding Agent Index: 80	Capital and Compute
Claude Fable 5	$50.00	$11.80	SWE-Bench Pro: 80.3%	Beri

When the Cheapest Agent Wins on Quality

The most provocative data point of the summer comes from TestSprite’s CoderCup, which ran four frontier AI coding agents through the same ten-phase build under identical rules. The cheapest agent built the most accurate application at half the cost of the priciest model in the field. Not half the cost with a quality tradeoff. Half the cost with better output.

Databricks ran a similar experiment on their own polyglot codebase — Scala, Go, Rust, Java, TypeScript, and more — and found that open-source GLM-5.2 matched top closed models on real engineering tasks. More striking: a minimal custom harness they built (called Pi) matched vendor harnesses at half the cost with the same success rate. Cheaper per-token models like Sonnet 5 actually cost more per completed task than Opus 4.8 despite lower per-token prices, because they consumed more tokens to reach the same outcome.

This is the pattern I keep seeing: capability gaps are marginal, cost spreads are extreme, and most teams adopt tools without measuring either. The Databricks team called per-token rate cards “nearly useless” for agentic workloads. They’re right. When a model that costs 4x less per token produces the same result in the same harness, the leaderboard rank is noise.

The implication for your procurement process is direct. Stop asking “which model is best?” and start asking “which model completes our tasks cheapest at acceptable quality?” Those are different questions with different answers — and the second one is the only one that shows up in your budget.

The Harness Problem: Same Model, Different Results

A model’s benchmark score depends heavily on the harness — the scaffolding around the model that handles tool calls, context management, error recovery, and test execution. Change the harness and you can shift scores dramatically without touching the model.

Consider the evidence. On the neutral kingy.ai benchmark, Grok 4.5 scores 83.3% on Terminal-Bench 2.1 and takes first place on SWE Marathon at 29%, per Pondero. But on DeepSWE 1.1, a test that demands more sustained multi-step reasoning, Grok drops to 53% against Fable’s 70% and GPT-5.5’s 67%. Same model, different harness demands, dramatically different results.

In JetBrains’ Kotlin Benchmark — 105 real engineering tasks sourced from active open-source repositories — Claude Code with Opus 4.7 xhigh resolved 85.71% (90/105), the top result. JetBrains Junie with Opus 4.7 max and Codex with GPT 5.5 xhigh followed at 81.9% each. The gap between first and third is under 4 percentage points. The harness and tool integration — not the raw model — drove most of the difference.

Cognition’s SWE-1.7 illustrates the same point from a different angle. It scores 42.3% on FrontierCode 1.1 Main, 81.5% on Terminal-Bench 2.1, and 77.8% on SWE-Bench Multilingual, per Cognition’s own report. Those are all self-reported numbers on benchmarks Cognition either created or selected. The model was trained from a Kimi K2.7 base that already scored 30.1% on FrontierCode — Cognition’s RL pipeline added 12.2 percentage points. That’s a real gain. But every score is from the lab that built the model, evaluated on a benchmark the lab designed.

The takeaway: when you see a benchmark score, you’re reading the output of a specific model in a specific harness evaluated by a specific entity. Change any of those three variables and the ranking can flip. AGENTS.md best practices matter here too — your harness configuration can swing outcomes more than your model choice.

Building a Verified Task Benchmark That Actually Predicts Spend

The Databricks approach is the model worth copying. They didn’t run a public benchmark and hope it generalized. They sampled their own codebase, built tasks that reflected their actual engineering work, and measured cost per completed task in a harness they controlled. The result was data that directly predicted their spend.

Here’s how to build one for your team:

Sample real tasks from your backlog. Pull 30–50 completed tickets that span your typical work — bug fixes, feature additions, refactors. Strip them into self-contained prompts with clear acceptance criteria.
Run each task through multiple models in the same harness. Use identical agent configuration, context windows, and tool access for every model. The only variable should be the model itself.
Measure cost per verified task, not cost per token. Track total token consumption (input + output), multiply by your actual rates, and divide by the number of tasks that pass your acceptance criteria. This is your verified task economics number.
Route work across a cheap executor plus an independent reviewer. Use a different vendor for review than the one that wrote the code, so the reviewer has no shared blind spots. GPT-5.5 reviewing Claude’s output, or vice versa, catches errors that same-vendor review misses.
Re-run quarterly. Model capabilities and prices shift monthly. A benchmark from March is stale by July. The 72-hour period between July 8 and July 10, 2026 saw four frontier coding models launch with a 12x price spread — your economics can change in a week.

The Databricks finding that a minimal harness halved cost at equal success rate is the most actionable data point here. You don’t need a vendor’s integrated agent harness. You need a thin orchestration layer that routes work to the cheapest model that can handle it, escalates to a stronger model when the cheap one fails, and runs an independent reviewer on the output. That architecture costs a fraction of a single-model subscription while producing equivalent or better results.

The Real ROI Question You Should Be Asking

How much does AI-assisted development actually save? The gap between perceived and measured productivity is the largest blind spot in engineering AI budgeting.

The DX data confirms this at scale. A median 7.76% PR throughput gain across 400+ organizations is real but modest. Most organizations land in the 5–15% range. The teams that come out ahead aren’t the ones with the most expensive model — they’re the ones who measured what worked, understood why it didn’t, and routed work accordingly.

Here’s the decision framework I’d recommend:

If you’re spending more than $200/developer/month on AI coding tools and can’t quote your cost per verified task, you’re flying blind. Build the internal benchmark first. Everything else is guessing.
If your harness is a vendor’s integrated agent with no custom orchestration, you’re paying 2x what you need to. The Databricks Pi harness proved a minimal custom setup halves cost at equal success rates.
If you’re routing all work to a single frontier model, you’re wasting budget on capability gaps that don’t exist. GLM-5.2 matched top closed models on Databricks’ polyglot codebase. Muse Spark 1.1 costs $0.26 per task at the same Intelligence Index score as models costing 3x more.
If your procurement decision rests on a leaderboard screenshot, you’re letting vendors grade their own homework. Run the models on your codebase, in your harness, with your acceptance criteria.

The question isn’t whether AI coding tools deliver value — they do, at 5–15% throughput improvement. The question is whether you’re capturing that value or burning it on the wrong model at the wrong price in the wrong harness. The teams that measure cost per verified task will spend a fraction of what the leaderboard-chasers pay, and ship the same code. Which team are you on?