Citing fears that AI systems may soon spiral out of human control and potentially kill all humans, two more researchers from Anthropic and Google DeepMind who recently left their coveted positions are sounding the alarm about risks from advanced AI systems.
uncertain
who → cite → systems
Joe Benton, who used to lead a safety research team at Anthropic, and Josh Engels, who used to work on AI safety research at Google, told NBC News in their first interviews since they left that they see an urgent need to boost transparency about incidents at the cutting edge of AI given the rapid pace of AI development.
Advances in AI research “could speed up the pace of progress from merely blistering at the minute to uncontrollable” rates of development, Benton said in an interview with Tom Llamas.
uncertain
Benton → use → Llamas
“There are no adults in the room,” Engels added.
asserted
Engels → be → room
“People are trying their best, but there is no one coming to save us.”
asserted
People → try → us
Benton and Engels spoke with NBC News in the wake of a viral social media post from former Anthropic researcher Jacob Coxon, who left Anthropic on Tuesday.
asserted
who → speak → Tuesday
In his post on X announcing his departure, Coxon highlighted his extreme concern about the pace of AI development and the risks he believes it poses to the future of humanity.
asserted
it → announce → humanity
The post has been viewed more than 155 million times, spurring calls from legislators to hold special sessions of Congress to take action and sparking a wave of AI employees to speak out in support of his concerns.
asserted
post → view → concerns
Benton and Engels both pointed to the recent cyberattack against AI startup Hugging Face, carried out in July by autonomous AI systems powered by an unreleased OpenAI model, as part of the reason for shifting their work now.
“If you look at some of the recent incidents, these were not cases where humans told the models to do something bad,” Engels said.
asserted
Engels → point → something
Instead, OpenAI’s AI systems autonomously decided to hack into Hugging Face’s systems, create a sort of illicit message board to exchange information and even expose some of OpenAI’s own computing infrastructure to the open internet.
asserted
systems → decide → internet
“The models decided that the best way to to accomplish their task was to commit really egregious actions, to commit crimes,” Engels said in an interview with Christine Romans.
asserted
Engels → decide → Romans
OpenAI said that it has since strengthened its safeguards and that newer public models, including its most recent Astra system, more reliably follow human instructions.
asserted
models → say → instructions
An Anthropic spokesperson said in a statement Wednesday: “We have always been transparent that AI will bring both enormous benefits and unprecedented risks.
asserted
AI → say → benefits
To address these risks, we continue to build models with some of the strongest safeguards in the industry.”
asserted
we → address → industry
Benton managed a group at Anthropic dedicated to creating ways for humans and weaker AI systems to supervise more capable AI systems.
asserted
humans → manage → systems
He said that he is particularly worried that the public does not have significant insight into how AI systems have already exceeded the bounds of human instructions and that the lack of transparency could only get worse as systems become more capable.
uncertain
systems → say → transparency
“At the minute, basically all of the transparency about these risks that is coming from the companies is entirely voluntary,” Benton told NBC News.
asserted
Benton → come → News
In a blog post released Wednesday, OpenAI’s head of global affairs, Chris Lehane, agreed that the status quo is insufficient.
asserted
quo → release → affairs
“Today, frontier laboratories largely set their own rules for managing frontier risks,” Lehane wrote.
asserted
Lehane → set → risks
“Democratically accountable standards, independent verification, and meaningful transparency would replace that fragmented system of private governance.”
No federal law mandates that the largest AI companies, like OpenAI and Anthropic, share reports when agents or AI systems act beyond humans’ control.
asserted
agents → replace → control
Benton and Engels are joining METR, one of the world’s leading AI safety nonprofit research centers, to work on investigations into incidents or episodes in which AI strays from human directions or intentions.
asserted
AI → join → directions
METR aims to create scientific ways to evaluate how AI systems could cause catastrophic risks and empower researchers and the public to sway their development.
uncertain
systems → aim → development
“I left because I think I can have more positive influence on the development of this technology by helping to foster public transparency from outside these companies and to shed light on the risks,” Benton said.
asserted
Benton → leave → risks
The researchers joined growing numbers of AI safety researchers raising awareness about today’s AI systems following Coxon’s post Tuesday.
asserted
researchers → join → post
Marcus Williams, an OpenAI employee who works on monitoring the activity of AI agents, wrote Thursday afternoon on X, “Unless there is AI regulation or a coordinated slowdown between labs, human extinction in the next few years seems very likely.”
Geoffrey Irving, who was the chief scientist at the United Kingdom’s AI Security Institute and served stints at Google and OpenAI, seemed to agree Wednesday on X.
asserted
who → work → X.
“I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years.”
asserted
all → think → years
AI researchers have referenced the possibility that AI systems could potentially begin to improve themselves without human input, creating the potential for AI systems to either purposely or incidentally kill humans.
uncertain
systems → reference → humans
Benton said he was most worried that the pace of technological improvement could lead to this sort of digital or artificial superintelligence, in which AI systems are more capable than humans across most tasks.
uncertain
systems → say → tasks
“All of these companies — and this is something I witnessed firsthand at Anthropic — are pretty directly trying to race towards automating the process of AI R&D itself,” he said.
asserted
he → witness → R&D
Benton suggested that there’s a chance AI systems could come to exist as a sort of separate species before long.
uncertain
systems → suggest → species
“We’re probably going to go from a world where we have very capable systems now to a world where potentially we are co-inhabiting a world with AI agents that are much, much smarter than humans at some point in the next few years,” he said.
asserted
he → go → years
Engels said many people outside the AI industry might not fully appreciate just how intelligent today’s AI systems already are — and how capable they might soon become.
uncertain
they → say → industry
“We’re building these systems that are generally intelligent,” he said.
asserted
he → build → systems
“They can generally do what people can do, and soon they might be able to generally do what people can do, but better.”
uncertain
people → do → what
Both researchers said they were excited to shift their attention to more public efforts to highlight the latest AI progress, so people can make more informed decisions.
asserted
people → say → decisions
“There are many reasons people are excited and racing forward, Engels said, citing huge potential upsides and benefits from AI.
asserted
Engels → be → AI
But he added that “we should progress as society aware of the risks and okay with where they’re at.”
asserted
they → add → where
“I am worried that stuff might end up progressing too fast for us to get our act together in time,” Benton said, “unless we worry about it now.”
uncertain
we → worry → it
The Aporia
