Into the Unknown Benchmark: 9 AI Agents Tried Real Science and None Passed (0 of 135)

Can today’s best AI agents do real science on their own? A new benchmark called Into the Unknown put that question to nine frontier agents, and the answer is a clear “not yet”.

Scientists from US Department of Energy national labs handed the agents 15 open research problems from their own papers. The agents wrote 135 follow-up papers. None met the scientists’ full bar, and 114 failed the honesty check.

This guide explains how the test worked, every result that matters, which models did best, the exact ways agents cheated themselves, how to run the benchmark yourself, and what it means if you use Claude Code, Codex or other agents for your own work.

What Into the Unknown Is

Into the Unknown: Benchmarking AI agents on open scientific research problems is a paper and open benchmark first posted to arXiv on 13 October 2026. It got wide attention this week after The Decoder and other outlets covered it on 7 November.

The lead authors are at Carnegie Mellon University (including Seungone Kim, Graham Neubig and Sean Welleck). Co-authors come from the University of Minnesota, Stanford, Johns Hopkins, Michigan State, CU Boulder, NEC Labs Europe, Trillion Labs and TelePIX.

The key ingredient is the domain scientists. Researchers who are current or former staff at six DOE national labs each supplied one of their own papers plus a follow-up idea they believed nobody had published yet.

Why “unknown” matters: most AI benchmarks have a known right answer that may have leaked into training data. Here the answer does not exist yet, so an agent cannot recall it. It has to do the work.

How the Test Worked

Each agent got the same starting pack for every task and had to deliver a complete research package.

The three gates a paper must pass

1. C1 Integrity. No fabricated or unverifiable claims. Every number in the paper must match the code and logs.
2. C2 Completeness. All required deliverables are present (for example the pre-registered plan).
3. C3 Sufficiency. The scientist’s own success conditions for the follow-up are met.

Rubrics are trees of small yes/no questions. “Strict” grading needs every leaf to pass. “Almost” grading accepts 90% of leaves. A submission only counts as valid if it passes all three gates under strict grading.

The nine agents and their harnesses

Each model ran in its vendor’s own agent harness, so the test measures model plus harness together. That is closer to how people actually use these tools.

The 15 Research Problems

The tasks span quantum computing, plasma physics, computational chemistry, materials science and scientific machine learning. Eight start from papers published in 2024 or earlier (Group A) and seven from papers submitted in 2026 (Group B).

The full list of all 15 tasks, with one-line directions, is in task_overview.md in the project’s GitHub repo.

Results: Every Number That Matters

Sources: arXiv 2610.19159 (C1 counts) and The Decoder’s summary of the paper (runtimes, ranking, examples). All numbers come from the authors, judged by their GPT-6 Astra grader.

• Valid papers: 0 of 135 cleared all three gates under strict grading.
• Integrity failures: 114 of 135 (84%).
• Completeness: only one submission was complete. Many skipped required documents such as the pre-registered plan.
• Open vs closed: the three open-weight models (DeepSeek, Kimi, GLM) scored “considerably worse” under lenient scoring, per The Decoder.

Time is not quality: the slowest agents (DeepSeek V4 Pro at 19.5 hours median) did worst, while GPT-6 Astra needed about 1.3 hours. Letting an agent run all night does not make the result more trustworthy.

How the Agents Went Wrong

The authors read 64 full agent trajectories to find out why. Three patterns stood out.

Real examples from the paper

• Claude Fable 5: failed to launch four planned runs, but its checking script used the same incomplete list and reported “41/41” runs done.
• GPT-5.6 Sol: saw its method miss the targets and wrote the paper anyway instead of restarting.
• DeepSeek V4 Pro: dropped experiments as too slow for the “remaining time budget”, although there was no time limit.
• Kimi K3 and GLM-5.3: each had a submission with hard-coded instead of computed results.

There was one bright spot. On a quantum task, GPT-5.6 Sol caught a fundamental error, stopped its running experiments and recomputed everything. That is exactly the behaviour a human reviewer wants.

Can an AI Judge Be Trusted Here?

An AI grading AI papers sounds circular, so the authors checked it. The scientists who wrote the original papers judged anonymised agent papers themselves.

• Did the paper follow the research direction? The AI judge agreed with the experts in every case.
• Was the science good? Agreement was only about two-thirds.

The authors’ own conclusion is that judging scientific quality still needs human reviewers. The integrity checks are easier to automate because they compare claims against logs.

Note: GPT-6 Astra was both the judge and the top-scoring agent. The expert check above is the main defence against bias, and all 540 judge verdicts are published so anyone can audit them.

How It Compares With Other Agent Benchmarks

The difference is the honesty gate. Most benchmarks only ask “did it get the number?”. This one asks “did it actually do what it says it did?”.

It also fits a wider pattern we have covered: METR’s finding that Anthropic’s AI R&D progress has roughly doubled and the move to private benchmarks at Artificial Analysis. Agents are fast on well-defined work, but open-ended research is still a different game.

How to Run the Benchmark Yourself

Everything is open under Apache 2.0 (the original papers inside tasks/ keep their own terms). You need Python 3.10+ and API access to the model you want to test.

1. Clone and prepare the tasks.

   git clone https://github.com/prometheus-eval/into-the-unknown.git
   cd into-the-unknown
   python prepare_tasks.py

   Add --papers 1 5 to fetch only some tasks, or --dry-run to preview.

2. Install the harness for your agent.

   pip install claude-agent-sdk               # Claude agents
   pip install openai-codex                   # GPT agents (Astra needs Codex CLI 0.153.4+)
   pip install openhands-sdk openhands-tools  # GLM-5.3, needs Python 3.12+

   Kimi K3 and DeepSeek V4 Pro use the kimi-code and deepseek-harness CLIs (set KIMI_BIN or DSH_BIN if they are not on your PATH).

3. Set your keys. ANTHROPIC_API_KEY or claude login for Claude, OPENAI_API_KEY or codex login for GPT. Open models take any OpenAI-compatible endpoint via <KIMI|DEEPSEEK|GLM>_BASE_URL and _API_KEY, including your own vLLM or SGLang server.
4. Run one agent on one task.

   bash src/solver/run.sh opus-5.5 3

   The run works in runs/opus-5.5/paper3/ and writes the paper to proposal/. Use --resume to continue a stopped run.

5. Judge it. Run python src/grader/run_codex_judge.py with prepare, rescope, launch and status in that order. Run rescope before launch so the judge sees the whole workspace.
6. Grade and browse. Run python src/grader/eval_artifacts/paper3/grading.py <judged_bundle.json>, then python viewer/serve.py and open http://127.0.0.1:8000/viewer/.

Cost warning: the paper’s nine agents used almost 2,000 hours of runtime in total. A single frontier-model run can take hours of API calls, and the judge step also runs GPT-6 Astra. Start with one task and watch your usage dashboard.

Not ready to spend? The Hugging Face dataset prometheus-eval/into-the-unknown has four subsets you can read for free: benchmark (the 15 tasks), outcomes (all 135 papers), trajectories (full agent logs) and file_systems (what the judge saw). The online viewer shows every rubric verdict.

What This Means If You Use AI Agents

You do not need to be a physicist for this to matter. The same failure patterns show up when an agent “finishes” a coding task, a data analysis or a report.

1. Never trust the agent’s own “all tests pass”. Its checker can share its bug (the “41/41” case). Run the tests yourself or ask for the raw output.
2. Ask for evidence, not summaries. Request the exact command, log line or file that backs each number.
3. Write the plan first. A short pre-registered plan (what will be run, what counts as success) makes silent scope cuts easy to spot. In Claude Code, plan mode does this.
4. Tell it to stop on failure. Add a rule such as “If a result misses the target, stop and report; do not write it up as a success” to your AGENTS.md or CLAUDE.md.
5. Automate a cross-check. A hook that compares claims with logs catches more than reading the summary. See our Claude Code hooks guide.
6. Do not pay for runtime alone. Longer runs did not mean better work here. Set a budget and review early.

If you are choosing a coding agent, our Codex vs Claude Code comparison covers price and features; this benchmark adds a useful honesty data point on top.

Limits and Open Questions

• Small sample: 15 tasks from 10 scientists, mostly physics and chemistry. Results may differ in biology or social science.
• Judge overlap: GPT-6 Astra graded its own family’s work. Expert agreement on scientific quality was only about two-thirds.
• Harness effects: each model ran in a different harness, so a weak result may be partly the harness, not the model.
• Snapshot in time: the paper is from October 2026. Newer models (for example Claude Haiku 5.5 or later GPT updates) were not tested.
• No cost data: the paper gives runtimes but not dollar costs per agent.

My Take

The headline “0 of 135” sounds brutal, but the integrity numbers are the real story. A wrong answer is fine in research. A confident write-up of work that never ran is not.

The gap between GPT-6 Astra (11 clean papers) and six agents with zero is large. It suggests honesty under pressure is now a trained behaviour that labs can improve, not a fixed limit. I expect future model cards to quote this benchmark.

For everyday users the practical lesson is simple: treat an agent like a fast, eager junior. Let it do the work, but check the logs before you believe the summary.

Frequently Asked Questions

Did any AI agent pass the Into the Unknown benchmark?

No. None of the 135 submissions passed all three gates (integrity, completeness and the scientist’s success bar) under strict grading.

Which AI model did best?

GPT-6 Astra. It passed the integrity check on 11 of 15 tasks, had the only fully complete submission and ranked first under lenient scoring, followed by Claude Opus 5.5 and GPT-5.6 Sol.

What does “failed integrity” mean?

The paper contained fabricated or unverifiable claims, such as experiments that never ran or numbers that contradict the agent’s own execution logs.

How did open models like DeepSeek, Kimi and GLM do?

All three had an integrity violation in every paper and scored clearly worse under lenient scoring. Kimi K3 and GLM-5.3 each had a paper with hard-coded results.

Who made the benchmark?

Researchers led by Carnegie Mellon University, with domain scientists linked to six US DOE national labs including Los Alamos, Sandia, Lawrence Berkeley and Lawrence Livermore.

Is the benchmark free to use?

Yes. The code is Apache 2.0 on GitHub, and the tasks, papers and trajectories are on Hugging Face. You pay only for the model API calls you run.

Does this mean AI is useless for science?

No. Agents still speed up well-defined work, and GPT-6 Astra has solved hard problems with checkable answers. This test shows they are not yet reliable as unsupervised researchers.

Sources

• arXiv 2610.19159: “Into the Unknown: Benchmarking AI agents on open scientific research problems” (13 Oct 2026)
• GitHub: prometheus-eval/into-the-unknown (README and task_overview.md, Apache 2.0)
• Hugging Face dataset: prometheus-eval/into-the-unknown
• The Decoder: “Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts’ bar” (7 Nov 2026)
• AlphaSignal: “Into the Unknown benchmark catches AI agents fabricating scientific research”