Skip to content
, ,

FrontierHarness Eval v1.1 Explained: Which Coding Harness Wins per Model

10–15 minutes
Woman in a server room aisle holding a card reading BENCHMARK, hook 5x cost gap, same model

Runta published FrontierHarness Eval v1.1 on 17 November 2026. It ran 22 coding harness setups with 5 open models across 27,280 runs. The big finding is simple: the best harness changes when you change the model.

This matters if you pick tools like Codex CLI, OpenCode or Claude Code. Most leaderboards rank models. This one asks how much the wrapper around the model changes your cost and pass rate.

TLDR

  • Scale: 22 harness configurations x 5 models x 62 terminal tasks x 4 attempts = 27,280 runs and 83B tokens.
  • Winners change per model: Codex CLI (GLM-5.3), Pi (GLM-5.3 Flash), ZCode (DeepSeek V4.1 Flash 1017), Oh My Pi (Muse) and Alma (Kimi-K3).
  • Model beats harness on cost: the model explains 92.8% of cost variance. The harness explains 2.3% and the pairing 4.9%.
  • Same model, big price gaps: the priciest harness cost 1.8x to 5.0x the cheapest per task.
  • Paying more did not buy more passes: cost and pass rate barely correlated.
  • Not tested: Claude, GPT and Gemini models. Claude Code, Cursor and Gemini CLI are not in v1.1.

What FrontierHarness Measures: Harness vs Model

A coding agent has two parts. The model writes the code. The harness is the program around it that reads files, runs shell commands, manages context and decides when to stop.

Codex CLI, OpenCode, Pi and Cline CLI are all harnesses. Many of them can talk to any model. So the question is fair: if you keep the model fixed, does the harness change the result?

FrontierHarness v1.1 asks two practical questions:

  • Should you switch harness when you switch model? In other words, is there one best harness for everyone?
  • Which harness is the best value for each model? Value here means the cheapest cost per passed task, with a quality guard.

Runta built the study with Baseten, which served all five models. The article is written by Shilin Zhu at Runta.

Quick answer: FrontierHarness holds the model fixed and swaps the harness. It shows that harness choice can change your bill by up to 5x on the same model, and that no single harness wins on every model.

How the Test Was Run

The v1.1 setup is about 76 times bigger than v1.0, by Runta’s count. Here are the main parts.

Item v1.1 setting
Harness configurations 22 (18 harnesses; 4 DeepSeek Harness profiles and 2 OpenCode versions count separately)
Models 5 open models: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash 1017, Muse (muse-spark-open), Kimi-K3
Tasks 62 terminal tasks, time limits from 15 minutes to 8 hours
Attempts 4 per harness, model and task
Total runs 27,280, using 83B tokens
Runner Harbor, default harness settings, maximum reasoning effort
Isolation A fresh Runta runtime per trial, deleted afterwards
Scoring Pass or fail by a verifier
Cost Estimated inference cost at list prices

How cost and value are calculated

Runta uses two cost numbers. Average cost per task is the mean cost of priced trials for each task, then averaged over all tasks. Cost per pass is that number divided by the pass rate.

The best-value rule picks the harness with the lowest cost per pass. There is one exception. If another harness costs at most 30% more per pass but passes at least 5 points more often, that one wins instead.

Cheating checks and robustness

  • Reward hacking: audits flagged 216 trials, 163 of them online lookups. All flagged trials scored 0.
  • Resampling: the 62 tasks were resampled 4,000 times to see how often each winner stays on top.
  • Error bars: each pass rate has a 95% interval of about plus or minus 10 points, so small gaps are ties.

The Verified Results

The full per-cell tables on Runta’s page are interactive charts. Their numbers do not appear in the article text, so this guide only lists figures that Runta states in writing.

Best-value harness for each model

Model Best-value harness What Runta reports Stays first in resamples
GLM-5.3 Codex CLI $0.803 per pass 99.6%
GLM-5.3 Flash Pi Lowest cost per pass 42%
DeepSeek V4.1 Flash 1017 ZCode Beat OpenCode v1: 25% more cost, 7.7 more points 47%
Muse Oh My Pi Lowest cost per pass 53%
Kimi-K3 Alma Beat Oh My Pi: 9% more cost, 6.5 more points 25%

Only one winner is solid. Codex CLI with GLM-5.3 stayed first in 99.6% of resampled draws. The other four won in 25% to 53% of draws, so a different task mix could crown a different harness.

Across all five models

Harness Verified figure
OpenCode v2 Lowest cost per task: $0.561 (on the cost/pass frontier)
Pi Highest pass rate: 61.9% (on the frontier)
Codex CLI Highest pass rate: 61.9%; $0.937 per pass overall
ZCode Highest pass rate: 61.9% (on the frontier)
Antigravity CLI $1.63 per pass overall
mini-SWE-agent Smallest first request (882 tokens), 38 steps per pass, 17th of 22 on cost per pass
All other harnesses Up to 1.6x the cost, 50.6% to 60.2% pass rate

Note: On DeepSeek V4.1 Flash 1017, pass rates across harnesses ran from 54.0% to 69.0%. That is the widest pass-rate spread Runta mentions for a single model.

Biggest Takeaways

1. Harness rankings do not carry over between models

The median harness moved 11 of 22 places in cost rank between models. Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3. DeepSeek Harness: Creator went from 1st with Muse to 21st with GLM-5.3 Flash.

Runta also split each model’s attempts into two halves. Cost rankings from the two halves agreed well (rank correlation 0.71). Rankings from different models barely agreed (0.26). So the reshuffle is real, not noise.

2. The model drives most of the cost

  • Model: 92.8% of the variance in cost per task.
  • Model-harness pairing: 4.9%.
  • Harness alone: 2.3%.

Pick the model first. Then tune the harness, because even 2.3% plus 4.9% can still mean a 5x bill gap on one model.

3. Same model, up to 5x the price

With the model fixed, the most expensive harness cost 1.8x to 5.0x the cheapest per task. Shuffled data only gave 1.2x to 1.8x, and every real gap beat every shuffle (p = 0.003).

The worst case was DeepSeek. Cline CLI cost 5.0x as much as OpenCode v1 for just 2.0 more points.

4. Spending more does not buy more passes

The rank correlation between cost per task and pass rate ran from -0.39 to +0.22 per model. In plain words, the pricier harness was not reliably the better one.

5. Steps matter more than prompt size

Steps per pass predicted cost per pass (+0.51, p = 0.015). Start tokens, tool count and cache hit rate did not (-0.03 to +0.02). A tiny system prompt does not help if the agent takes many turns.

6. Most money goes to failed runs

  • 64.5% of list-price cost went to runs that did not pass: 52.7% to failures and 11.8% to timeouts.
  • Among the five best-value harnesses, 68% to 82% of failed trials ended with a confident final answer, not a timeout.
  • Cost split across all trials: cached input 42%, output 33%, other input 25%.
  • Average input per trial: 2.0 to 4.8 million tokens.

Warning: A confident “done” from an agent is not proof. Most failed runs in this study ended that way, so always run your tests before you merge.

What the winning harnesses share

Runta lists these as observations, not tested causes:

  • One shell tool: a single shell tool handled 61% to 93% of tool calls.
  • Append-only context: each request only added to the last one, so most input hit the cache.
  • Few extras: subagents and summaries were rare.

What v1.0 Said About Claude Code

Claude Code is not in v1.1. It was in v1.0, published on 1 September 2026, which tested 12 configurations with one model: Kimi K3 on Fireworks. That run used 30 tasks (21 from Terminal-Bench, 9 from DeepSWE), one attempt each, for 360 runs.

Harness (v1.0) Pass rate Median cost per pass Median time
Codex 66.7% $3.47 6m 43s
DSH Creator 63.3% $3.28 6m 44s
Claude Code 63.3% $18.34 9m 38s
Pi 60.0% $2.43 7m 33s
DSH Minimal 56.7% $4.72 5m 41s
Exo Harness 53.3% $1.05 6m 17s
OpenCode 50.0% $3.24 6m 27s

Claude Code tied for second on passes but had the highest cost per pass. Runta said part of that is harness-model fit, since Claude Code was running a non-Claude model. Its token-weighted cache hit rate was only 25.0%.

Tip: v1.0 and v1.1 use different tasks, models and attempt counts. Do not compare their dollar figures side by side.

FrontierHarness vs Artificial Analysis Coding Agent Index v2.1

Artificial Analysis released its Coding Agent Index v2.1 on 13 November 2026. Both studies score harness plus model, but they answer different questions.

Point FrontierHarness v1.1 AA Coding Agent Index v2.1
Main question Which harness is best value for a given model? Which agent setup scores highest overall?
Models 5 open models only Closed and open models (Claude, GPT, Gemini and more)
Harness pairing Every harness with every model Mostly each vendor’s own agent
Tasks 62 terminal tasks 322 tasks across 3 benchmarks
Attempts 4 per task 3 per task
Headline metric Cost per pass and pass rate Index score plus cost and time per task
Serving One provider (Baseten) Pay-per-token API prices

There is a small overlap. AA lists OpenCode with GLM-5.3 (Max) at 40 points and $5.41 per task. It lists DeepSeek Harness with DeepSeek V4.1 Flash 1017 (Max) at 37 points and $0.32 per task.

FrontierHarness tells you those same models can be cheaper per pass in other harnesses. For example, Codex CLI was the best-value pick for GLM-5.3. Use AA to choose a model and FrontierHarness to tune the wrapper.

What It Means for Claude Code, Codex, Cursor, OpenCode and Gemini CLI

  • Codex CLI: the strongest showing. It tied for the top pass rate (61.9%) and won GLM-5.3 with high confidence. It ranked 16th on cost with Kimi-K3, though. See our Codex vs Claude Code guide.
  • OpenCode: v2 had the lowest cost per task overall ($0.561). OpenCode v1 was the cheap baseline on DeepSeek. Compare it in OpenCode vs Claude Code.
  • Claude Code: not in v1.1. With Claude models, AA’s index is the better guide. With third-party models, v1.0 hints it can be costly per pass.
  • Cursor: not tested in either FrontierHarness version. See Cursor vs Windsurf vs Claude Code vs Cline for how it compares.
  • Gemini CLI: not named in v1.1. Google’s Antigravity CLI was tested and cost $1.63 per pass overall.
  • DeepSeek Harness: four profiles were tested, and Creator swung from 1st to 21st. Read our DeepSeek Harness guide before choosing a profile.

How to Read Harness Benchmarks: A Step-by-Step Check

  1. Check the model list. If your model is not in the study, the harness rankings may not apply. v1.1 has no Claude, GPT or Gemini models.
  2. Find your model’s row, not the overall row. Rankings moved 11 places on average between models.
  3. Look at the error bars. A gap under about 10 points in pass rate is a tie in v1.1.
  4. Check the resample win rate. 99.6% is a clear winner. 25% means the winner is fragile.
  5. Compare cost per pass, not cost per task. A cheap harness that fails often can cost more per finished job.
  6. Match the pricing to yours. v1.1 uses list prices. Subscriptions, discounts and cache pricing change the maths.
  7. Test on your own repo. Runta’s own advice is to try candidate harnesses on your tasks before you switch.

My Take

This is one of the most useful harness studies so far. It shows something many developers feel but rarely measure: the same model can be cheap in one tool and costly in another.

The limits are real, though. All five models are open and served by one host. Runta says it plans to add Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon next, and that version will matter more for most Claude Code and Codex users.

For now, my advice is to pick your model first, then try two or three harnesses on your own repo. Watch cost per finished task, not the sticker price.

Common Problems and Fixes (Misreadings)

“Codex CLI is the best harness, full stop”

Fix: it won only with GLM-5.3. With Kimi-K3 it ranked 16th on cost. Check the row for your model.

“Claude Code did badly in FrontierHarness”

Fix: Claude Code is not in v1.1 at all. In v1.0 it tied for second on pass rate but ran a non-Claude model with high cost per pass.

“The harness barely matters because it is only 2.3% of variance”

Fix: that 2.3% is across all models. Inside one model, harness choice still moved cost per task by 1.8x to 5.0x.

“A 3-point lead means the harness is better”

Fix: with about plus or minus 10 points of uncertainty, a 3-point lead is a tie.

“The pricier harness must be smarter”

Fix: cost and pass rate showed no reliable link (-0.39 to +0.22). Cline CLI cost 5x OpenCode v1 on DeepSeek for 2 points.

“These dollar figures match my bill”

Fix: they are list-price estimates on Baseten at maximum reasoning effort. Your provider, plan and effort setting will change the numbers.

Frequently Asked Questions

What is FrontierHarness Eval v1.1?

It is a Runta and Baseten study published on 17 November 2026. It tested 22 coding harness configurations with 5 open models on 62 terminal tasks, for 27,280 runs.

Which harness won FrontierHarness v1.1?

No single harness won. The best-value pick was Codex CLI for GLM-5.3, Pi for GLM-5.3 Flash, ZCode for DeepSeek V4.1 Flash 1017, Oh My Pi for Muse and Alma for Kimi-K3.

Is Claude Code included in FrontierHarness v1.1?

No. Claude Code appeared in v1.0, where it scored 63.3% with Kimi K3 at a $18.34 median cost per pass. v1.1 does not list it.

Does FrontierHarness test Claude, GPT or Gemini models?

Not yet. v1.1 used only open models served by Baseten. Runta says it plans to add Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon.

What is cost per pass?

It is the average cost per task divided by the pass rate. It shows what you pay, on average, for each task the agent actually solves.

How is it different from the Artificial Analysis Coding Agent Index?

AA ranks agent setups, mostly vendor agents with their own models, by an overall score. FrontierHarness keeps the model fixed and compares harnesses on cost per pass.

Where can I see the full FrontierHarness results?

The interactive tables are on Runta’s v1.1 blog post and the frontierharness.org dashboard. Runta says the job data is stored on Harbor Hub.

Sources

Free weekly email

Get new AI guides every week

One short weekly email with our newest local AI, ComfyUI and AI tool guides. No spam, no pop-ups, just the good stuff.

Free. Unsubscribe anytime in one click. We only use your email to send this newsletter. See our privacy policy.

Woman in a server room aisle holding a card reading BENCHMARK, hook 5x cost gap, same model