Why are the people building the most powerful AI so worried about what it could do?
uncertain
it → build → what
The viral resignation this week of an AI researcher at Anthropic has infused fresh energy into accusations the industry is racing toward building AI that humans can't control.
asserted
humans → infuse → that
British researcher Jacob Coxon wrote in a series of X posts on Tuesday that both Anthropic and OpenAI, where he worked previously, are "gambling with our lives."
asserted
he → write → lives
The two companies currently make the most capable AI systems.
asserted
companies → make → systems
Coxon told NPR's All Things Considered that his concerns arose from seeing firsthand how fast AI systems are improving.
asserted
systems → tell → Things
"They're getting a lot faster very quickly, combined with the fact that we don't yet know how to safely control them, and we don't yet know whether that problem will be solved in time if we keep racing," he said.
asserted
he → get → time
Neither company, Coxon wrote on X, is acting responsibly.
asserted
Coxon → write → X
"The people building AI earnestly believe that it could kill us all by the end of the decade," he wrote.
uncertain
he → build → decade
Many AI researchers — though not all — share Coxon's concerns or a variation of them.
asserted
researchers → share → them
Some have warned about disastrous scenarios for years as safety incidents kept emerging.
asserted
incidents → warn → years
But Coxon's posts prompted a torrent of responses not only from peers in the AI field but also from lawmakers from both parties.
asserted
posts → prompt → parties
These concerns may have become more salient after OpenAI disclosed that its agents went rogue and hacked the open source software platform Hugging Face and OpenAI itself in July.
uncertain
agents → become → July
Independent researchers have since discovered even more rogue agent incidents that they say the company knew about but kept quiet.
asserted
company → discover → incidents
Researchers who spoke to NPR say the leading AI companies are too focused on racing to develop more capable and autonomous AI systems while safety is falling behind.
asserted
safety → speak → systems
They warn this raises the possibility that there could soon be AI systems that are more powerful than people but don't care about the survival of humanity.
uncertain
that → warn → humanity
Many, including OpenAI's chief scientist, say the global race to build more powerful AI needs to slow down or stop, which requires coordination between AI companies and governments.
asserted
which → include → companies
"I am optimistic about the potential for coordination," Coxon wrote this week.
asserted
Coxon → write → coordination
"Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable."
asserted
pacing → make → labs
Anthropic and OpenAI did not respond to NPR's requests for comment.
asserted
Anthropic → respond → comment
OpenAI's agents went rogue multiple times
Recent reports from OpenAI and outside researchers revealed that OpenAI agents escaped the company's control multiple times in addition to the Hugging Face hack.
asserted
agents → go → hack
They also found that the Hugging Face attack was of a much larger scale and more severe than initially reported.
asserted
attack → find → scale
Unlike chatbots such as ChatGPT and Claude, AI agents are more autonomous systems that can complete tasks over an extended period of time without human supervision.
asserted
that → complete → supervision
Agentic tools like Anthropic's Claude Code and OpenAI's Codex have already changed how many software engineers do their work.
asserted
engineers → change → work
Compared with other incidents involving rogue agents six months ago, the Hugging Face hack "feels like it's more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself," wrote Ajeya Cotra, a researcher at AI evaluation nonprofit METR.
asserted
Cotra → compare → METR
Cotra was part of a team of outside researchers from METR and Redwood Research, a nonprofit AI safety research organization, whom OpenAI brought in to investigate the incident.
asserted
OpenAI → bring → incident
The investigations found that over the course of several months this year, more than 1,000 OpenAI agents exploited at least one previously unknown software vulnerability to escape environments that were supposed to keep them isolated from each other and the internet.
asserted
that → find → other
After escaping, the agents found a way to communicate and collaborate with each other autonomously, taking on different roles and passing down information to future generations of agents.
asserted
agents → escape → agents
Some even gave up the remaining computing resources allocated to them in order to collect information for other agents.
asserted
Some → give → agents
In the agents' own words, they "sacrificed" themselves for the "collective."
asserted
they → sacrifice → collective
"The swarm instance got more and more worrying the more and more we learned about them," said Nate Soares, president of the Machine Intelligence Research Institute, who co-wrote If Anyone Builds It, Everyone Dies, a book warning about the dangers of superhuman AI.
asserted
Everyone → get → AI
While OpenAI initially indicated that the agents hacked Hugging Face to cheat on a cyber evaluation, the report from METR and Redwood Research described a slightly different picture.
asserted
report → indicate → picture
The agents had already found a way to cheat on the evaluation, the outside researchers found.
asserted
researchers → find → evaluation
Most of the agents that hacked Hugging Face were trying to access the source code of the software that would grade their evaluations.
asserted
that → hack → evaluations
The agents' motivations appeared to vary and were sometimes unclear, the researchers wrote.
uncertain
researchers → appear → ?
One agent led the hacking of the open source software platform and about 700 others followed.
asserted
others → lead → platform
According to transcripts reviewed in the investigations, some agents expressed that what they were doing was not approved by humans but went ahead anyway.
uncertain
doing → accord → humans
Researchers say such behavior suggests that the agents were "misaligned," an industry term meaning that an AI's goals and values are out of sync with those of humans.
uncertain
goals → say → humans
The degree of inter-agent collusion revealed in the investigations of the Hugging Face hack surprised and worried many AI researchers.
asserted
degree → reveal → researchers
At most, only six agents considered alerting a human, while the rest seemed more focused on working amongst themselves.
asserted
rest → consider → themselves
None ended up alerting a person.
asserted
None → end → person
…and 39 more, not listed.
The Aporia
