Artificial Analysis Goes Private: Open Models Now 12 Points Behind

Artificial Analysis has rebuilt its Intelligence Index, and 60% of the new version now runs on private test sets or held-out answers.

The result is a much bigger gap between open-weight models and the closed leaders: from 3 points to 12 at the top, and to 26 points for models whose weights you can download today. If you pick models from leaderboards, this change matters for you.

What changed in Intelligence Index v5

Artificial Analysis is an independent group that runs the same set of tests on many AI models and combines the results into one number, the Intelligence Index. Many people use it as a quick way to compare models from Anthropic, OpenAI, Google, DeepSeek, Moonshot and others.

On October 27, 2026, the team released version 5. They call it the biggest upgrade since v4 launched in January. Here is what moved:

• New tests: FrontierCode 2.0, a private coding benchmark from Cognition that grades whether a project maintainer would actually merge the change, and Terminal-Bench-Science, a set of 70 agentic science research tasks run in a terminal.
• Upgraded tests: HLE-Diamond replaces Humanity’s Last Exam with a 1,000-question version split evenly between reasoning and knowledge. AA-Omniscience moves to v1.1. CritPt v2 adds a private 40-question hard subset.
• Kept: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0.
• Removed: GDPval-AA v2.1, SciCode and AA-LCR v1.1.

The index now gives equal weight (25% each) to four areas: knowledge work, coding, science and general ability.

The team also launched a separate Progress Index v1, which places every language model released since November 2022 on one scale so you can see progress over time. The Intelligence Index stays the main measure of how smart current models are.

Why private tests matter: the contamination problem

To understand why this update shook the rankings, you need one idea: benchmark contamination.

A public benchmark is a list of questions and answers posted online. AI models learn from huge amounts of web text. Over time, those questions, the answers, and even blog posts discussing them end up in training data.

A model can then score well because it has seen the test, not because it can solve new problems. It is like a student who memorized last year’s exam paper.

There is a softer version too. Labs can tune a model hard toward the exact style of popular public tests. Reporters have called this “benchmaxxing.” The model looks great on the leaderboard but feels weaker on your real work.

Private test sets fix much of this. If the questions never leave the testing team, no model can train on them.

Held-out answers work in a similar way: the questions may be public, but the correct answers are not, so a model cannot simply copy them.

With 60% of v5 built this way, the score is closer to “how well does this model handle problems it has never seen?”

Two of the private parts are named directly by Artificial Analysis. FrontierCode 2.0 keeps its tasks private, and the CritPt v2 hard subset is private. AA-Omniscience has a private set plus a newly released public subset, which was recalibrated so the two give consistent results.

Private tests do have one downside. You cannot check the questions yourself, so you have to trust the testing team. That is why it still makes sense to run a few of your own prompts before you commit to any model. More on that in the steps below.

The new rankings at a glance

Here are the headline scores from the v5 launch article. “Max” and “high” refer to the reasoning effort setting used for the test.

A few details stand out from the per-test results:

• FrontierCode 2.0: Claude Opus 5.5 leads at 46%. GPT-6 Astra, GPT-6.1 Terra and Claude Sonnet 5.5 sit at 42%. The best open-weights result is DeepSeek V4.1 Flash at 28%.
• Terminal-Bench-Science: GPT-6 Astra leads at 63%, then Claude Opus 5.5 at 59%. Kimi K3.1 Pro gets 34%.
• HLE-Diamond: Claude Opus 5.5 leads at 68%, GPT-6 Astra has 64%, Kimi K3.1 Pro has 37%.
• CritPt v2 hard subset: GPT-6 Astra leads at 72%. Kimi K3.1 Pro is the top open model at 45%.
• Terminal-Bench 4.0: Kimi K3.1 Pro actually leads this one at 67%. Open models are not weak everywhere.

The biggest mover was Kimi K3.1 Pro. Press coverage reports it fell from fourth place under v4 to ninth under v5. For a full list of where each model stands this month, see my best AI model leaderboard for November 2026.

Why the open-weights gap grew

Under v4.3, the best open-weights model trailed the leader by only 3 points. Under v5 it trails by 12. Among models whose weights could be downloaded on launch day, the gap is 26 points (32 against 58).

There are a few likely reasons, and it is worth being careful here:

1. Private tests remove any benefit from public test exposure. If any model gained from seeing public questions, that advantage is gone. Artificial Analysis does not claim that any specific model was contaminated. The change is a general safeguard.
2. The new mix favors long agentic work. Agentic tests (Terminal-Bench-Science, Terminal-Bench 4.0, AA-Briefcase and FrontierCode 2.0) make up 55% of the weight. Median output tokens per task rose from 35k in v4.3 to 59k in v5. Frontier closed models have been tuned hard for long, multi-step tasks.
3. Stricter grading. In FrontierCode 2.0 a task that fails any blocking check scores zero. In Terminal-Bench-Science a task passes only if every test passes. Partial credit is gone.

Note: Coverage of the launch makes an important point: a rank change after a method change is not the same as a model getting worse. Kimi K3.1 Pro is the same model it was a week earlier. The ruler changed, not the object.

Also note the “open” label on Kimi K3.1 Pro. At launch its weights were not yet public. Moonshot had said they would follow on November 4. If you need to self-host, check the model’s official Hugging Face page before you plan around it.

Cost, effort settings and token use

The v5 article also shows how much the effort setting and price can change the picture:

• Cost per Intelligence Index task: Claude Opus 5.5 (max) $10.04, GPT-6 Astra (max) $6.24, GPT-6.1 Sol (max) $1.27.
• Among the top 10 models, cost varies by more than 13 times. GPT-6.1 Terra (max) is cheapest at $0.96 per task, Claude Fable 5.1 (max) is the most expensive at $13.04.
• Claude Opus 5.5 scores 35 at low effort (about $0.98 per task) and 58 at max effort. That is a huge spread.
• Claude Sonnet 5.5’s gap between low and max effort widened to 29 points (from 20). GPT-6.1 Sol’s gap doubled from 10 to 20.

The lesson: the same model can behave like two different products depending on the effort setting. When you compare scores, compare at the setting you will actually use. For API price details, see my AI API pricing comparison and the Claude API pricing guide.

How to choose a model now (step by step)

A leaderboard is a starting point, not a final answer. Here is a simple process you can follow.

1. Write down your main task. Is it coding, document work, research, chat support, or data extraction? Each index area maps to different work, so pick the one that fits.
2. Check the per-test results, not only the headline number. For coding, look at FrontierCode 2.0 and Terminal-Bench 4.0. For science or research, look at Terminal-Bench-Science and CritPt. For factual accuracy, look at AA-Omniscience, which also rewards models for not making things up.
3. Decide if you need open weights. If you must run the model on your own servers (privacy rules, no internet, fixed costs), limit your list to models whose weights are actually released. Today that means the 32-point tier, not 46.
4. Match the effort setting to your budget. Look at cost per task. A model at 55 for $1.27 may beat a model at 58 for $10.04 when you run thousands of calls.
5. Shortlist two or three models. Include one strong closed model and one cheaper option.
6. Run your own private test. Take 10 to 20 real examples from your work that have never been posted online. Run each model on them. This is your personal private benchmark, and it is immune to contamination.
7. Score the results simply. Mark each answer pass or fail, note the cost, and note how long it took. Pick the model with the best mix for you.
8. Re-check every few months. Benchmarks and models change quickly. A model that wins today may not win in February.

If you want to try many models with one API key while you test, an aggregator like OpenRouter can help. My OpenRouter guide explains how it works.

My Take

I think the move to private tests is good news for everyone who picks models for real work. Public benchmarks were becoming too easy to game, even by accident. A score that leans on unseen problems is a more honest signal.

That said, I would not read this as “open models are bad now.” Kimi K3.1 Pro still leads Terminal-Bench 4.0, and the released open models at 32 are close to Claude Haiku 5.5 at 36.

For many everyday jobs like summaries, extraction and simple chat, a 32-point open model you control can be the right call. The gap matters most for hard, long, agentic tasks, where the top closed models clearly pull ahead.

Tip: My advice is simple: use v5 to build your shortlist, then let your own private test make the final choice.

Common Problems and Fixes

Problem: “My favorite model dropped, so it must have gotten worse.” Fix: The model did not change; the test did. Re-run your own prompts. If it still works well for you, there is no need to switch.

Problem: Comparing scores from v4.3 and v5 side by side. Fix: Do not mix versions. The test list and weights changed, so numbers from different versions are not comparable. Use only v5 numbers for current decisions.

Problem: Ignoring the effort setting. Fix: Check which setting (low, high, max) produced the score. Claude Opus 5.5 scores 35 at low and 58 at max. Test at the setting you plan to use.

Problem: Planning to self-host a model whose weights are not out. Fix: Confirm the weights are on the official Hugging Face or GitHub page, and check the license, before you buy hardware.

Problem: Costs explode after switching to a top model. Fix: Long agentic runs use many tokens. Set token limits, use lower effort for easy tasks, and use prompt caching. My prompt caching guide shows how.

Problem: “I can’t see the private questions, so how do I trust it?” Fix: You can’t check them, and that is the point. Balance it by running your own small private test on real tasks.

Frequently Asked Questions

What is the Artificial Analysis Intelligence Index?

It is a single score that combines many tests run by the independent group Artificial Analysis. It covers knowledge work, coding, science and general ability, each weighted at 25% in v5.

When did Intelligence Index v5 launch?

Artificial Analysis published it on October 27, 2026, alongside a new Progress Index v1.

Which model is number one in v5?

Claude Opus 5.5 at max effort leads with 58. GPT-6 Astra (max) is second at 57, and GPT-6.1 Sol (max) is third at 55.

How big is the open-weights gap now?

The best open-weights model, Kimi K3.1 Pro, scores 46, which is 12 points behind the leader. It was only 3 points under v4.3. For models with weights released at launch, the best score is 32, which is 26 points behind.

What is benchmark contamination?

It is when test questions or answers leak into a model’s training data. The model then scores well because it has seen the test, not because it can solve new problems. Private test sets prevent this.

Does this mean open-source models are not worth using?

No. They are behind on hard agentic tasks, but they are cheap, private and fully under your control. For many simple tasks they are a good fit. Test them on your own work.

How often should I re-check the rankings?

Every few months, or whenever a major model launches. New models arrive often, and Artificial Analysis updates scores as it tests them.

Sources

• Artificial Analysis: Intelligence Index v5 launch article (October 27, 2026)
• Business Insider: Chinese AI models’ rankings after Artificial Analysis switched to private tests
• OfficeChai: Intelligence Index v5 with majority private datasets
• Let’s Data Science: Index v5 drops Kimi K3.1 Pro from fourth to ninth
• Let’s Data Science: Kimi K3.1 Pro API, open weights due November 4