Anthropic Researcher Resigns Over AI Loss-of-Control Risk, Safety Controversy Heats Up Ahead of IPO
nashnova research
Anthropic researcher Jacob Coxon quit over fears that AI is racing beyond control, just as the company pursues a $2 trillion IPO valuation — This means → the firm built on the promise of safety can't retain its own safety talent, raising questions about the valuation story the market has bought into.
Why did this researcher walk away?
Coxon is 27, a mathematician who worked on training new AI models with large-scale data.
His core judgment: without government intervention or coordinated industry slowdown, no single company can responsibly build AGI — artificial general intelligence, meaning AI that outperforms humans across many tasks.
He says many peers now use words like "crunch time" and "endgame" to describe the trajectory of self-improving models — AI systems that optimize themselves and accelerate on their own.
"By the end of next year, things could already be out of control." This means → he believes the window to hit the brakes may be less than 18 months.
Why couldn't he stay at "the safest company"?
Coxon left OpenAI earlier this year specifically for Anthropic, known for its safety focus. He still chose to leave.
He acknowledged Anthropic's safety work is genuine — but the problem isn't any single company's attitude. It's the competitive structure of the entire industry.
In plain terms = when every rival is accelerating and Chinese startups are adding pressure, safety compromises become nearly inevitable. One company's sincerity can't change the direction of the race.
He's not alone: another Anthropic safety researcher quit earlier this year to study poetry, warning that "the world is in danger." OpenAI has seen similar departures in recent years.
What are industry leaders themselves saying?
OpenAI chief scientist Jakub Pachocki published a blog post calling for coordinated slowdown and government intervention.
OpenAI CEO Sam Altman warned at last week's G20 summit: "Unless urgent action is taken, there will be serious problems in cybersecurity."
Anthropic CEO Dario Amodei has repeatedly warned about rogue AI models and called for slower development.
This reflects a contradiction: industry leaders call for the brakes while each company keeps accelerating. In plain terms = foot on the brake in speeches, foot on the gas in the lab.
What is government doing about it?
The U.S. currently has no federal AI regulation. The Trump administration prioritizes light-touch oversight to maximize AI's economic upside.
Senator Bernie Sanders and Rep. Greg Casar introduced a bill last week to permanently ban superintelligence and pause model development until regulators write new rules.
Coxon, Pachocki, Amodei, and over a thousand AI researchers recently signed a joint statement urging governments to coordinate a "brake mechanism" for self-improving models.
This means → the internal outcry is loud, but the gap between the pace of legislation and the pace of AI development remains vast.
What does this mean for Anthropic's IPO?
Anthropic has long built its brand on "responsible AI development" and used that positioning to scale a massive AI platform.
The question now: when safety talent starts voting with their feet, can that brand story still support a $2 trillion valuation?
This signals something deeper for the market: the premium investors pay for AI companies rests largely on the assumption that "these companies can govern themselves" — and that assumption is being shaken by insiders' own actions.
市场有风险，内容仅供研究参考，不构成投资建议。