Co-Design Bench Explained: Alibaba’s Coding Agent Benchmark Where the Best Model Finished Only 26.4% of Tasks

Alibaba researchers have released Co-Design Bench, a coding-agent benchmark built from real user requests. The best model, GPT-6 Astra, met 69.9% of requirements on average but fully completed only 26.4% of tasks.

That gap says a lot about how far “almost done” is from “done”. Here is how the benchmark works, the full leaderboard, and what it means for your own agent workflow.

What Co-Design Bench Is

Most coding benchmarks hand an agent a clean, well-written task. Real users do not work like that. They ask for a change, look at the result, then ask for fixes.

Co-Design Bench tries to capture that messy reality. Its paper is titled “Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models”.

• Authors: Alibaba’s Token Hub group with Fudan, Zhejiang, Shanghai Jiao Tong, NTU and CUHK.
• Published: arXiv 2610.35138, submitted October 26, 2026. It is a preprint, not peer reviewed yet.
• Focus: editing existing web apps, not writing code from scratch.
• Reference-free: no gold code or target screenshots. Only hidden acceptance criteria.

How the Tasks Were Built

The team started with a huge pile of real usage data and filtered it hard.

They favoured requests where users had to rephrase or roll back changes, since those signal that an agent struggled. Tasks were also ranked by structural difficulty and how easy they were to verify.

The two tracks

• Multimodal Editing (100 tasks): local visual, layout and content edits. Some include image, audio, video or document inputs, such as pulling an amount from a receipt.
• Repository Editing (173 tasks): coordinated changes across components, routes, state and data flow in multi-file frontend apps.

Each task starts from a runnable app recovered from the session logs, plus a cleaned-up version of the user’s request.

How Grading Works: Judge Then Refute

Grading uses two AI roles, a method the paper calls “judge-then-refute”.

1. The judge: grades each criterion as pass, partial or fail, using the source code and evidence from a real browser.
2. The refuter: a second LLM challenges every positive verdict. It hunts for stubs, hardcoded outputs, unreachable features and mismatched requirements.
3. Evidence rules: visual claims need renders and screenshots. Interaction claims need recorded browser actions. Missing evidence aborts the evaluation.

The three scores

• Strict completion: the average share of criteria fully passed per task.
• Graded completion: same, but partial passes count as half.
• Full-pass rate: the share of tasks where every single criterion passed. This is the 26.4% headline.

All ten models ran in the same Claude Code harness, with fixed containers, prompts and tools. Each run was capped at 3 hours and 120 turns.

Note: two frontend experts spot-checked 200 outputs. Swapping in different judge models moved absolute scores by at most about 0.05.

The Full Leaderboard

Here are all ten endpoints from the paper, ranked by combined strict score. “Full” is the share of tasks fully completed.

Note which models are missing. Newer releases such as Opus 5.5, Fable and Grok 4.7 were not in this run, so the table is a snapshot, not a final ranking. For a wider view, see my AI model leaderboard for November 2026.

Five Findings Worth Knowing

1. Partial is not done

GPT-6 Astra passed about 70% of criteria, yet 73.6% of its tasks still had at least one unmet requirement. Agents get most of the way, then miss a detail.

2. Multi-file work is much harder

Every model scored far lower on Repository Editing. The top full-pass rate drops from 41.0% on multimodal tasks to 17.9% on repo tasks.

3. Combined changes hurt everyone

All ten models did worse on repo tasks mixing three or four edit types than on tasks with one or two. Pooled strict scores fell from 0.654 to 0.591.

4. More tokens did not mean better results

GPT-6 Astra used about 1.24 million tokens per multimodal task. GLM-5.2 used about 7.29 million and still scored lower.

5. A small open model punches above its weight

Qwen3.8-27B ranks fifth on multimodal edits but second on repository strict score. If you run models locally, my Qwen3.8 27B on a 16 GB GPU guide shows how.

How to Run Co-Design Bench Yourself

The code is on GitHub at Wiselnn570/Co-Design-Bench, and the public data is on Hugging Face. Only the 96-task multimodal subset (521 criteria) is public so far. The repository track is not released yet.

1. Check requirements. You need Python 3.10 or newer. The reference judge also needs an authenticated Claude CLI, Node.js, Playwright and Chromium.
2. Install the package. From the repo root, run python -m pip install -e ..
3. Install the browser tools. Run npm install --no-save playwright and npx playwright install chromium.
4. Try the demo. Run cdb list, then score the bundled example. It should report Strict 66.67%, Graded 75.0% and Full pass 50.0%.
5. Get the dataset. Follow the repo’s dataset guide. The data archives are distributed separately from the code.
6. Run your agent. Save each finished app under submissions/<case_id>/.
7. Prepare, judge and score. Use the three commands below.

cdb prepare --dataset /path/to/evaluation-inputs --submissions /path/to/submissions --out work/evaluation
cdb judge --cases work/evaluation/cases.json --out results/run-1 --model opus
cdb score --cases work/evaluation/cases.json --results results/run-1/journal.jsonl --out results/run-1/scores.json

Warning: agents build and run real apps during evaluation. Run everything in a disposable environment, and expect real API costs for both the solver and the judge.

Limitations and Caveats

• Preprint: the paper is not peer reviewed yet.
• Frontend only: tasks are web apps. Backend, systems and data work are not covered.
• One harness: all models ran in Claude Code. Results in Codex or other agents may differ.
• AI judges: grading is done by LLMs, checked by human spot checks rather than full human review.
• Partial public data: outsiders can only reproduce the multimodal subset today.
• Snapshot: several current frontier models are not in the table.

What It Means for Your Coding Agent

If your agent says “done”, it is probably mostly done. This benchmark suggests you should expect at least one missed requirement on most real tasks.

• Write criteria up front: list acceptance checks before the agent starts. A plan step like Claude Code plan mode helps.
• Verify in a browser: the benchmark only trusts screenshots and recorded clicks. You should too.
• Split big changes: mixed multi-type edits failed most. Break them into smaller tasks.
• Watch for fakes: the refuter looks for stubs and hardcoded values for a reason. Check them in review.

My Take

The 69.9% versus 26.4% gap is the most honest number I have seen about coding agents this year. It matches what many developers feel: agents are fast, but the last 10% still needs you.

It also echoes other recent results, like the Into the Unknown benchmark where no agent passed real science tasks. Treat agents as strong juniors, not finishers.

Troubleshooting: Common Problems and Fixes

• cdb command not found: rerun python -m pip install -e . in the repo root and make sure your Python scripts folder is on PATH.
• Judge fails to start: the reference judge needs an authenticated Claude CLI. Log in first, then retry.
• Browser evidence missing: install Chromium with npx playwright install chromium. Missing evidence aborts a case by design.
• Scores do not match the paper: the paper used full infrastructure and the private repo track. Public runs cover the 96-task subset only.
• Submission not found: check each app sits in submissions/<case_id>/ using the fixed case IDs.
• App fails to build: the reference runner supports static frontends and npm builds only.

Frequently Asked Questions

What is Alibaba’s new coding agent benchmark called?

It is called Co-Design Bench. The paper is arXiv 2610.35138, from Alibaba’s Token Hub group and five universities.

Which model topped Co-Design Bench?

GPT-6 Astra. It passed 69.9% of criteria on average and fully completed 26.4% of tasks.

How did Claude do?

Claude Opus 5 came second, with 66.1% of criteria passed and 18.3% of tasks fully completed. Newer Claude models were not tested.

How many tasks are in the benchmark?

There are 273 tasks with 2,331 hidden criteria. The public subset has 96 multimodal tasks and 521 criteria.

Why is the full completion rate so low?

A task only counts as complete if every criterion passes. Agents often miss one detail, especially on multi-file changes.

Is Co-Design Bench open source?

The evaluation code is on GitHub and a public data subset is on Hugging Face. The repository editing track is not public yet.

Which agent harness was used?

All ten models ran inside the Claude Code harness, capped at 3 hours and 120 turns per task.

Sources

• arXiv 2610.35138: Co-Design Bench paper
• Co-Design Bench project page and leaderboard
• GitHub: Wiselnn570/Co-Design-Bench
• The Decoder: AI coding agents still need human cleanup
• DEV Community: Alibaba measured it