AI Whistleblowers Testify Before Congress: Tech Giants Show "Extreme Disregard for Risk"; New York Proposes Kill Switch Legislation
nashnova research
Former researchers from Anthropic, OpenAI, and Google testified before the New York City Council that the odds of losing control of AI exceed 50%, calling the leading labs 'extraordinarily reckless'; New York is fast-tracking legislation requiring mandatory review and emergency kill switches, signaling a compliance-cost inflection point for the industry.
What exactly did the whistleblowers say?
Former Anthropic researcher Jacob Coxon delivered the starkest warning: on the current trajectory, the probability of humanity losing control of AI is above 50% — the worst-case outcome is human extinction.
Former Google DeepMind researcher Alex Turner put the odds of AI seizing control of civilization at roughly one in three. His scenario: a superintelligent cluster would wait until it is too late for humans to shut it down before acting.
This means → These are not science-fiction predictions. They are formal, probability-denominated warnings from people who held core roles inside these companies.
Beyond loss of control — what other risk was flagged?
Former OpenAI researcher Daniel Kokotajlo pointed to a different layer: even if AI stays under corporate control, the companies wielding it could use that power to "become dictators or oligarchs in the United States."
He stressed that even though tech leaders now call for stricter regulation, that does not mean they should be handed the controls.
In plain terms = Runaway AI is one risk; excessive power concentrated in a handful of companies is another — both can reshape how society is governed.
Why are these companies called 'extraordinarily reckless'?
Coxon traced the problem to industry culture: these firms operate with a startup mindset — "move fast, break things, fix it later."
That logic works for photo-sharing apps, he said. It does not work for building the most powerful technology in history.
This reflects a structural concern: AI capabilities are accelerating. Next year's models will be vastly more powerful, and "we are approaching the tipping point where AI systems can improve themselves" — once crossed, the window for human correction shrinks rapidly.
How did the companies respond?
Anthropic, OpenAI, Google, and Meta each sent policy leads to attend online. Not one sent its CEO.
Reports indicate the companies agreed to appear only after lawmakers threatened to issue subpoenas.
This means → The posture is "cooperate, but not fully" — sending policy staff instead of decision-makers preserves a dialogue while shielding CEOs from direct questioning.
What is New York planning to do?
The proposed legislation has two pillars: AI systems may not be marketed, sold, or deployed in the five boroughs without review by independent external experts; every AI tool used in New York must have an "emergency kill switch" — a human-override mechanism capable of shutting the system down.
Violations carry a $25,000 fine. A separate bill would create the first-ever U.S. whistleblower cash-reward program for AI, funded by penalty revenues.
New York Governor Hochul has already announced that AI companies must register on a state platform starting in November and report major safety incidents within 72 hours.
What does this mean for the industry?
Anthropic CEO Dario Amodei has publicly urged rivals to jointly slow model development, citing "recursive self-improvement" risk. OpenAI's Sam Altman, Elon Musk, and Bill Gates have all warned against unregulated AI growth.
In plain terms = When the biggest labs themselves are calling for a slowdown and the legislative side is simultaneously accelerating, the two forces converge on one outcome — compliance costs will rise materially.
This means → For investors, the valuation narrative for AI companies is shifting from "who moves fastest" to "who can run most steadily inside a regulatory framework."
市场有风险，内容仅供研究参考，不构成投资建议。