Senior figures linked to Anthropic have issued some of the starkest warnings yet about advanced artificial intelligence, arguing that its rapid development could pose an extreme danger to humanity. They are calling for slower, better-coordinated progress and enforceable emergency controls, while critics say such warnings may also serve commercial and regulatory interests.
Key takeaways
- Former Anthropic researcher Jacob Coxon said some AI staff are “genuinely frightened” by the pace of progress.
- Anthropic co-founder Jack Clark said mandatory, independently verifiable AI shutdown systems may be needed.
- Researchers and executives disagree sharply over the likelihood of human extinction.
- US lawmakers have proposed legislation requiring companies to disable dangerous AI systems.
The debate sits at the centre of the issues Power AI Media tracks across technology, policy and cybersecurity: how can society benefit from increasingly capable tools without allowing their risks to outpace oversight?
Researchers warn of a race against time
Coxon, who previously worked at OpenAI and Anthropic, said colleagues feared that uncontrolled progress could produce catastrophic consequences. He argued that any slowdown would need international coordination, including with China, to prevent a competitive race from undermining safety.
One scenario discussed by Anthropic chief executive Dario Amodei involves networks of autonomous bots acting together with enormous computing power and potentially taking control of internet-connected systems. Coxon suggested that such a development could become realistic within six months to a year, although these predictions remain contested.
Anthropic backs stronger safeguards
Amodei has said AI development itself should continue, but that companies and governments need time to address serious risks. Anthropic says it tests its models aggressively, publishes safety findings and has developed a framework for reducing the possibility of “AI misalignment” — systems pursuing outcomes that conflict with human intentions.
Clark told the BBC that most leading laboratories already have ways to stop their systems, but questioned whether voluntary arrangements would be enough. He said lawmakers may eventually need to require a shutdown mechanism that can be checked by an independent third party.
What a mandatory kill switch could mean
A reliable kill switch would be intended to disable an AI model or agent if it behaved dangerously. The practical details remain unresolved, including who would control the mechanism, how it would be verified, and whether an advanced system could bypass it or replicate itself elsewhere.
US lawmakers have proposed a “Kill Switch Act” that would require companies to shut down or limit problematic AI tools and give some government agencies authority to demand action. For Power AI Media readers following AI tools and cybersecurity, the proposal highlights how technical safety features are increasingly becoming matters of public policy.
Critics question the warnings
Not everyone accepts the most alarming forecasts. Hugging Face chief executive Clement Delangue and Grindr chief executive George Arison have argued that extinction claims may be exaggerated or shaped by companies’ commercial incentives. Anthropic is reportedly preparing for a major public offering, while the industry’s enormous valuations depend partly on expectations that AI will transform work and entire sectors.
US President Donald Trump has rejected calls to slow development, describing fears that AI will destroy humanity as a hoax and arguing that the United States must maintain its lead over China. The dispute therefore combines technical uncertainty with geopolitics, investment and competition.
An unresolved policy challenge
Even sceptics acknowledge that powerful AI systems deserve scrutiny. The central question is no longer simply whether AI can deliver advances in healthcare, productivity and content generation, but whether safeguards can be tested and enforced before capabilities become harder to control.
For audiences across Power AI Media’s news, analysis and wider AI library, the warning from Anthropic’s figures is best understood as a call for evidence-based governance — not proof that an AI catastrophe is inevitable.