AI companies like Anthropic and OpenAI have long mentioned the possibility that their products could be existentially dangerous, but this has recently crescendoed into even more dire warnings, including suggestions that we have to pace the frontier, ie coordinate to slow AI progress.
uncertain
we → mention → progress
In response, cynical commentators have suggested this is a conspiracy to boost the companies’ stocks.
uncertain
this → suggest → stocks
The exact motives imputed vary from person to person.
asserted
motives → impute → person
Some say the companies are hyping the product by talking about how dangerous it is; if AI is smart enough to kill all humans, it must be smart enough for your business workflow.
asserted
it → say → workflow
Others say they’re trying to distract from near-term harms; every inch of newspaper space devoted to human extinction is one less inch that can be spent on plagiarism and water use.
asserted
that → say → plagiarism
But the most popular story involves regulatory capture: if the product is unsafe, maybe it can only be trusted to a few big politically-connected companies, and everyone else should be banned.
asserted
everyone → involve → companies
Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false.
asserted
all → resolve → stories
Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
asserted
who → affiliate → ones
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
asserted
industry → give → industry
The AI Experts Who Aren’t Affiliated With Big Companies Are Just As Concerned As The Ones Who Are
Grace et al surveyed 1,580 AI researchers who presented papers at big AI-related conferences, mostly academics with no company affiliation.
asserted
who → affiliate → affiliation
Over half thought there was at least a 10% chance AI would cause “human extinction or similarly permanent and severe disempowerment of the human species”.
asserted
AI → think → species
72% said we needed to be prioritizing AI safety research more.
asserted
we → say → research
Geoffrey Hinton, Yoshua Bengio, and Yann LeCunn are collectively considered “the godfathers of AI” for inventing deep learning; in 2018, they received the Turing Award, sometimes called the Nobel Prize of computer science, for their accomplishment.
asserted
they → consider → accomplishment
In 2023, Geoffrey Hinton quit his job at Google to work full-time raising awareness of the potential existential dangers of AI; he said there was “between 10% and 50% chance” that his invention would destroy humanity.
asserted
invention → quit → humanity
In the same year, Yoshua Bengio also switched to activism, saying that he “regretted his life’s work” and that AI could have a “catastrophic impact”.
uncertain
AI → switch → impact
Neither was affiliated with a major AI company.
asserted
Neither → affiliate → company
(the third godfather, Yann LeCunn, runs a $3 billion AI company, and says he isn’t worried)
Along with these independent researchers, there are many others who formerly worked at companies, then quit to whistleblow about the dangers, or left to pursue anti-AI activism full-time.
asserted
who → run → activism
Paul Christiano left OpenAI in 2021 to found an AI alignment nonprofit; he later said there was a “50/50 chance of doom shortly after you have AI systems that are human level.”
asserted
that → leave → systems
Daniel Kokotajlo quit OpenAI in 2024, refusing to sign a non-disparagement clause even though it could have cost him $2 million in equity, in order to start a nonprofit warning about risks from AI (he says he thinks there’s about an 80% chance AI kills everyone).
uncertain
AI → quit → everyone
Jacob Coxon quit Anthropic in September, also likely giving up millions of dollars in equity, because he thought it was “gambling with our lives”.
asserted
it → quit → lives
The people at AI companies are no more likely to raise the alarm about AI dangers than people who were never at AI companies, or people who were previously at AI companies but have since quit.
asserted
who → raise → companies
The AI Company Leaders Giving These Warnings Were Concerned Long Before They Ever Had AI Companies.
asserted
They → give → Companies
DARIO AMODEI first started discussing the risk of superintelligence destroying humanity in 2013, before OpenAI or Anthropic was founded, and when Amodei wasn’t employed in tech at all.
asserted
Amodei → start → tech
Here’s a 62 page debate between Amodei, Eliezer Yudkowsky, and other AI safety enthusiasts about strategy, dated October 27, 2013:
asserted
debate → ’ → strategy
In 2017, when Amodei was an ordinary employee at OpenAI, he spoke about his work the on the podcast 80,000 Hours (July 21, 2017):
asserted
he → speak → podcast
I mainly work on safety.
asserted
I → work → safety
We have a safety team that’s so far, myself and Paul Christiano.
asserted
that → have → team
Paul was co-author of mine on the [Concrete Problems In AI Safety] paper and has also written a lot online on his blog about AI.
asserted
Paul → write → AI
He’s probably one of the people who’s I think done the most to promote clear thinking about the problem and tie it to current AI.
asserted
I → ’ → AI
We have a third person joining in a few weeks who I’m super excited about.
asserted
I → have → who
We’re trying to build up a team that focuses on technical safety.
asserted
that → try → safety
We also do a little bit of strategy stuff which is how do we get different organizations that are working on AI to cooperate with each other, how do we cooperate with policy makers on questions like these.
asserted
we → do → these
We’re also thinking a little bit about those issues but mainly technical safety.
asserted
We → think → issues
Dario expanded on the cooperation point - he told the host about his fear that AI would lead to an “arms race” in which companies competed to develop the technology first and let safety slide.
asserted
safety → expand → technology
His solution was making companies cooperate:
asserted
companies → make → ?
We all think about AGI and think, that safety issues are important. . . then the question is, how does that scale to there being a lot of organizations?
asserted
that → think → organizations
does that scale to others who get involved once they see how powerful AI is?
asserted
AI → scale → others
Can we make them cooperate as well?
asserted
them → make → ?
This became an obsession with him, and he hired an advisor to help OpenAI work on the problem.
asserted
OpenAI → become → problem
In 2017, Amodei hired Page Hedley, a former public-interest lawyer, to be OpenAI’s policy and ethics adviser.
asserted
Amodei → hire → Hedley
…and 89 more, not listed.
The Aporia
