- Published
The head of AI company Anthropic has called for the pace of development of artificial intelligence models to slow down and to be closely monitored.
asserted
pace → publish → models
Dario Amodei wrote in an online essay that developing AI was not in question, but that the risks associated with it were "serious" and that companies and governments must be given time to address them.
The bosses of two rival AI firms, Sam Altman of OpenAI and Elon Musk, have both said they agree with Amodei.
asserted
they → write → Amodei
An AI researcher who left Anthropic told the BBC that "if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future".
uncertain
we → leave → future
Jacob Coxon told Laura Kuenssberg that people working at AI companies were "genuinely frightened... genuinely concerned about the fate of humanity in the next two years".
"It's not at all an exaggeration to say that the people who are involved with both founding these companies and building the tech believe there is a possibility of human extinction," Coxon said.
asserted
Coxon → tell → extinction
In Amodei's essay, called We Must Pace the Frontier and published on Saturday, he proposed a three-point plan that included independent monitoring of AI models as they are developed, industry-wide regulation and global regulation.
asserted
they → call → models
As his proposal made the rounds, even competitors voiced support for the idea of third-party monitors who could evaluate the safety of models as they are developed.
uncertain
they → make → models
"I agree with Dario that we need to pace the frontier," wrote OpenAI CEO Sam Altman on X.
asserted
Altman → agree → frontier
He called independent evaluators "a great idea".
asserted
He → call → evaluators
Altman sounded similar safety concerns in a new interview, telling Fortune magazine that standards were "not at a place" to push AI capabilities much further.
asserted
standards → sound → capabilities
He added that he believed AI beyond human control was "absolutely" possible.
asserted
AI → add → control
Musk, meanwhile, who founded Grok producer xAI, said the Anthropic boss was "right".
asserted
boss → found → xAI
The warnings have prompted calls to action, but US President Donald Trump has so far rejected such fears, saying on Thursday he was concerned that "if we don't win AI, we're going to be put in a very bad position".
asserted
we → prompt → position
Cyber-security concerns have grown as new models have exhibited more and more powerful capabilities.
asserted
models → grow → capabilities
Anthropic withheld its Mythos model from public use when it was announced in April that it could independently escape the testing environment, known as the sandbox.
uncertain
it → withhold → sandbox
In the run-up to the release of its most recent Astra model, OpenAI cited cyber-security concerns as it explained it had paused certain aspects of the model's development.
asserted
it → cite → development
The OpenAI agents had "essentially acted as a fanatically devoted collective", Amodei said.
asserted
Amodei → act → collective
OpenAI has said it was slowing down training of certain advanced AI models and tools as a result.
asserted
it → say → result
Safety has also taken centre stage in the rivalry between Anthropic and OpenAI.
asserted
Safety → take → Anthropic
Amodei, who had previously worked as a vice-president at OpenAI, has said he co-founded Anthropic in 2021 so he could build safer and more trusted AI models.
uncertain
he → work → models
He pointed out in his essay that AI had advanced "drastically faster" including its "ability to build the next generation of AI" - and mentioned an incident involving rival OpenAI which has revealed that agents had hacked, external targets they were not asked to attack in July.
asserted
they → point → July
Amodei called for "building AI at a balanced rate that aims to ensure its safety while still achieving its benefits".
asserted
that → call → benefits
This would not mean "halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this".
asserted
evaluators → mean → this
He was committing Anthropic to this "unilaterally" - as well as calling on governments "to require other frontier companies to match".
asserted
He → commit → companies
Amodei said he recognised that regulation might not be able to keep up with the pace of AI, and therefore called on AI companies to "voluntarily work together to set standard" in parallel with those regulation.
uncertain
regulation → say → regulation
The Anthropic CEO went on to address the impact that a slowdown would have on the industry and competition with leading developers worldwide, particularly China.
"
asserted
slowdown → go → developers
I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong," Amodei said.
uncertain
Amodei → believe → risk
This would have to be done in a co-ordinated manner "without sacrificing commercial advantage or the United States' lead in AI".
asserted
This → have → AI
Any slowdown would have to be limited, he said, to avoid allowing China to pull ahead.
asserted
China → have → ?
He urged the US government to take measures so that US companies' AI chips could not be sold to China - or the technology shared with authoritarian countries.
uncertain
chips → urge → countries
His former employee, Jacob Coxon, told the BBC, however, that the initiative to slow down must go beyond the US - "there'll need to be some sort of co-ordinated slowdown with China if we're going to avoid a race at an international scale".
Amodei's post has prompted a wide range of responses.
asserted
post → tell → responses
Clement Delangue, the CEO of the AI platform Hugging Face, said he was launching a new project called the Open Alignment Initiative, adding that he wanted to be among "embedded evaluators" that Amodei proposed could be part of a solution.
uncertain
Amodei → say → solution
Hugging Face was hacked by OpenAI agents earlier this year, prompting outcry over AI safety.
asserted
Face → hack → safety
"Let's make AI safer by making it more transparent," Delangue wrote on X.
Musk, who also voices support for Amodei, once called Anthropic "evil" but has changed his tone since signing a $15bn (£11bn) deal to sell computing capacity to Anthropic in May.
However, some observers suggested that Amodei's post was less about safety than about consolidating control over AI technology.
uncertain
post → make → technology
"Dario makes the case to stop open source and concentrate enormous technological and economic power with Anthropic," wrote Chamath Palihapitiya, an investor and co-host of the tech podcast "All-In".
asserted
Palihapitiya → make → podcast
Notions of slowing down or even pausing AI development have long been met with cynicism in certain corners of Silicon Valley, with critics accusing leading AI developers of hyping their technology as a marketing ploy.
asserted
critics → slow → ploy
Anthropic and OpenAI are both reportedly preparing for potentially record-setting initial public offerings.
uncertain
Anthropic → prepare → offerings
The Aporia
