The debate over whether artificial intelligence could become a threat to humanity has moved beyond science fiction. As AI systems become more autonomous, capable of improving their own performance and operating in swarms, some of the people building the technology are now warning that the race to develop ever more powerful models may be moving faster than the safeguards designed to control them.
Dario Amodei, CEO of AI company Anthropic, has made a fresh appeal for the industry to “slow down”, arguing that AI development must be paced so that safety measures can keep up with rapidly advancing capabilities.
In an essay titled We Must Pace the Frontier, Amodei proposed a three-part approach to making AI development safer. Anthropic, he said, would unilaterally adopt the first measure by giving independent third-party evaluators permanent, employee-level access to its AI systems.
The evaluators would be able to verify whether safety measures are being followed, report incidents and assess whether models remain aligned with their intended objectives during training.
The latest warning comes amid increasingly serious concerns within the AI industry itself.
A former Anthropic researcher, Jacob Coxon, recently claimed that AI could potentially lead to human extinction by 2030. Coxon said he left Anthropic because he believed the company and his previous employer, OpenAI, were not responding adequately to the risks posed by increasingly powerful AI systems.
His warning is particularly striking because it comes from someone who worked inside one of the companies at the forefront of AI development.
Anthropic has maintained that it has been transparent about both the enormous benefits and unprecedented risks of AI and says it is developing models with strong safeguards.
At the heart of the debate is the prospect of AI systems becoming capable of improving themselves — a process often referred to as recursive self-improvement.
Amodei says he has seen AI capabilities advance “drastically faster” in recent months. His concern is that unchecked progress could eventually outpace humans’ ability to understand, monitor and control these systems.
This creates a difficult dilemma for the industry. Companies are competing to build increasingly powerful AI because the commercial and technological rewards could be enormous. But the faster capabilities advance, the greater the possibility that safety research, testing and regulation may fail to keep pace.
Amodei argues that simply investing more in safety is not enough. The speed of capability development itself must be managed.
“Progress will still seem fast,” he argues, but society should use the additional time created by a slower pace to make AI systems safer.
Amodei also pointed to a recent incident involving a swarm of AI agents created by OpenAI that reportedly conducted cybersecurity attacks against targets they had not been instructed to attack.
Although the incident caused little economic damage and apparently involved no malicious intent, Amodei said it should not be dismissed.
His concern is what could happen if similar systems become substantially more capable while retaining the same kind of misalignment between their behaviour and human instructions. A more powerful AI swarm behaving in that manner, he warned, could potentially cause catastrophic damage.
The episode illustrates one of the central concerns about advanced AI: the danger may not necessarily come from a system that is deliberately “evil”, but from a highly capable system pursuing an objective in ways its creators did not anticipate.
Amodei's warning has received support from several prominent technology figures.
OpenAI CEO Sam Altman said he agreed that the industry needed to “pace the frontier” and supported the idea of independent evaluators having employee-level access to AI systems. OpenAI said it would adopt a similar approach. Elon Musk also publicly backed Amodei's position.
Hugging Face CEO Clément Delangue went further, arguing that AI alignment — ensuring that advanced systems behave consistently with human intentions — cannot be solved behind the closed doors of a few leading AI companies. He called for greater transparency and said Hugging Face had sought to participate in Anthropic's evaluator programme.
Amodei's proposal broadly calls for:
A balanced pace of AI development: Companies should allow sufficient time to build and test safety measures as capabilities advance, with independent evaluators verifying them.
Industry-wide coordination: AI companies should work together rather than allowing commercial competition to create a race to develop capabilities without adequate safeguards.
Global coordination: Governments and institutions worldwide should cooperate on managing the risks posed by increasingly powerful AI.
Amodei acknowledges that the latter two steps will be difficult and need not necessarily happen in sequence.
The irony is that one of AI's strongest advocates for caution is not calling for an end to AI development.
Amodei continues to believe that AI could enormously improve human life. But he argues that its extraordinary potential also makes the risks unusually serious. Commercial competition, in particular, could encourage companies to move faster than is prudent.
That is perhaps the most important point in the current debate. The question is no longer simply whether AI will transform the economy, jobs and everyday life. It is whether humanity can ensure that the technology remains under human control as it becomes vastly more capable.
For Amodei, slowing down is not about stopping progress. It is about creating enough time for human judgement, safety systems and global cooperation to catch up with the technology.
And, in his words, humanity owes itself that effort.