New AI Benchmark Method Released: Gap Between Chinese Open-Source Models and U.S. Widens to 12 Points
nashnova research
After AI benchmarking firm Artificial Analysis made its test questions secret, the gap between top open-weight models and closed-source leaders widened from 3 points to 12 — suggesting the competitive threat from Chinese AI has been significantly overstated.
Why did the score gap blow open the moment tests went secret?
Artificial Analysis overhauled its evaluation system, making a large share of test questions confidential to stop AI companies from training specifically on known prompts.
Under the old system the best open-weight model trailed the overall leader by just 3 points; under the new system the gap jumped to 12 points.
This means → much of the "nearly caught up" story rested on targeted drilling on public questions, not genuine capability gains.
How far ahead are closed-source models, really?
The top open-weight score under the new system was just 32, while Anthropic's Claude Opus 5.5 scored 58.
Even Anthropic's smaller model, Claude Haiku 5.5, beat every open-weight model — including Moonshot's Kimi K3.1 Pro.
In plain terms = the closed-source camp didn't need its best player to win — even its second string outscored open source's top seed.
What is "benchmark cheating"?
Snowflake CEO Sridhar Ramaswamy asked on X: "Is open source catching up, or are test questions leaking?"
Microsoft CEO Satya Nadella and other industry figures call this "benchmark cheating" — models see the exam in advance, train on it, post high scores, then underperform in real-world tasks.
This reflects a deeper problem: many "open source is closing the gap" comparisons cited across the industry were built on a compromised testing foundation.
How much are open-source models actually being used?
OpenCode co-founder Dax Raad analyzed traffic on model aggregation platform OpenRouter and estimated that all DeepSeek traffic could run on a single rack of Nvidia high-performance servers — hardware cost roughly $6–7 million.
For contrast: ChatGPT alone has over 1 billion monthly active users, all running OpenAI's closed-source models.
This means → the market attention Chinese open-source models receive is wildly out of proportion to their actual compute footprint.
Why does this matter to investors?
The benchmark gap plus the usage gap point to one conclusion: the narrative that Chinese AI is rapidly closing in on U.S. leaders needs revisiting.
That reassessment directly affects the valuations of OpenAI and Anthropic — especially as Anthropic prepares for an IPO and OpenAI may eventually list.
In plain terms = if the competitive threat has been overstated, these two companies' moats may be wider than the market currently prices in.
市场有风险，内容仅供研究参考，不构成投资建议。