Claude's Top Model Mythos 'Blows Away' METR Evaluation Scale

A trend chart is sparking widespread discussions in the AI circle - not because the data is astonishing, but because the evaluation agency admits: existing tools can no longer keep up with the evolution of models.

METR's Conclusion: 16 hours, but with an exceptionally wide confidence interval

In March 2026, AI safety evaluation agency METR conducted a risk assessment on Anthropic's early version of Claude Mythos Preview, giving a "50% task completion timeline" of at least 16 hours, with a 95% confidence interval ranging from 8.5 to 55 hours.

The meaning of this indicator is: on tasks that human experts need to spend X hours to complete, the model has a 50% probability of completing them independently. Among METR's 228 test tasks, only 5 were categorized as 16 hours and above, making the measurement results of this interval "unstable and meaningless".

METR immediately noted on its website: "Measurement results above 16 hours are unreliable under the current task set." In other words, the accuracy of this number itself is questionable, but the directional signal it reveals is real: the existing evaluation framework has reached its upper limit.

Capability leaps are real, but need to be interpreted correctly

Looking at the historical trajectory, GPT-4o's 50% timeline in mid-2024 was about 7 minutes, Claude Opus 4.6 and GPT-5.2 clustered in the 5 to 6-hour range, while Mythos Preview fell above 16 hours, exceeding all publicly comparable models. The doubling cycle of the task completion timeline for cutting-edge models is about 105 days, with an annual growth rate of over 1000%.

On the software engineering benchmark SWE-bench Verified, Mythos scored 93.9%, leading other publicly available models by more than 13 percentage points; on the more difficult SWE-bench Pro, it led GPT-5.4 by 20 percentage points.

However, cognitive scientist Gary Marcus warns that this number needs to be interpreted cautiously. A 50% success rate is a relatively lenient threshold - if the requirement is raised to 80% or 95%, there is still considerable room for differentiation in the current evaluation set. In addition, METR's tasks are highly concentrated in the field of software engineering, which essentially differs from the work in the real workplace that involves communication coordination, organization knowledge, and ambiguous goals.

In the security domain: defensive and offensive capabilities are improving in tandem

Cybersecurity company Palo Alto Networks found after gaining early access to cutting-edge models such as Mythos that AI systems are increasingly capable of independently identifying software vulnerabilities and connecting dispersed low-risk vulnerabilities into complete attack paths. Their tests showed that vulnerability analysis work completed in three weeks with AI assistance is equivalent to a top-level penetration testing team's workload for an entire year.

On the defensive side, there is also evidence. Mozilla used Mythos Preview to scan the Firefox browser and repaired 423 security vulnerabilities in just April 2026, far exceeding the previous average of 17 to 31 per month, including a 20-year-old XSLT vulnerability and a race condition that could lead to sandbox escape.

The lag of evaluation infrastructure is the deeper issue

The greater significance of METR's evaluation this time may not lie in how strong Mythos' capabilities are, but in that it reveals a systemic problem: the evolution of cutting-edge models has begun to exceed the construction speed of third-party evaluation tools. METR said it is developing new evaluation methods that include longer-cycle tasks, but they are not yet completed.

Capabilities are accelerating, but the yardstick has not yet been created - this may be the most noteworthy reality in the current stage of AI development.