A week ago, Jacob Coxon resigned from his job at Anthropic.
asserted
Coxon → resign → Anthropic
Coxon was a researcher working on training the next generation of AI models.
asserted
Coxon → work → models
He had recently moved to Anthropic from OpenAI, where he had worked for years, because Anthropic has a reputation for being more safety conscious.
asserted
Anthropic → move → reputation
Yet he found that even Anthropic is rushing forward with the development of unsafe AI models.
asserted
Anthropic → find → models
And so Coxon walked away from his cushy job, where he stood to make millions in salary and stock options, to turn whistleblower.
asserted
he → walk → whistleblower
In a post on X, Coxon said that there is a chance that AI will lead to human extinction.
asserted
AI → say → extinction
Other researchers soon chimed in to say that the chance is at least 10 percent.
asserted
chance → chime → ?
Coxon’s post quickly went viral, in part because it was widely shared by his colleagues working in AI, many of whom agreed with his assessment of the magnitude of the risk.
asserted
many → go → risk
And leaders of AI companies apparently agreed.
asserted
leaders → agree → companies
This weekend, Dario Amodei, CEO of Anthropic, put out a statement endorsing the plan to “Pace the Frontier,” meaning to dramatically slow down the rate of development at “frontier labs.”
asserted
Amodei → put → labs
Sam Altman, CEO of OpenAI, quickly agreed, as did Elon Musk, whose xAI program is no slouch either.
asserted
program → agree → OpenAI
They agree with Coxon: AI is dangerous.
asserted
AI → agree → Coxon
Let’s try to put that 10% number into context.
asserted
’s → let → context
That’s a little higher than the odds that an NFL team wins the Super Bowl after not having a winning record the previous season.
asserted
team → ’ → record
It’s a mistake to try to put a precise number on these kinds of predictions.
asserted
It → ’ → predictions
We really don’t know what’s going to happen, and putting a number on things lends a fake air of scientific precision to what is essentially guesswork.
asserted
what → know → precision
But the comparison to a losing team winning the Super Bowl the next year should give you a good gut feel for what Coxon, Amodei, Altman, Musk, and the many others working in AI think about their products.
asserted
Coxon → lose → products
In the fury of discussion that has followed Coxon’s resignation, by far the most common objection I’ve seen from skeptics is that it seems fantastical: “How, exactly, might AI kill us all?”
uncertain
AI → follow → us
Some claim that AI companies are playing up the risks as a form of marketing: “Juicing their valuations” ahead of these companies’ expected public offerings.
uncertain
companies → claim → offerings
But the skeptics, I’ve come to believe, are wrong.
asserted
I → come → ?
We no longer live in the same world we did 18 months ago.
asserted
we → live → world
I had been familiar with the AI risk arguments for years, and found them interesting, albeit unconvincing.
asserted
them → find → years
But I began to take these arguments more seriously when agentic AI, capable of “vibecoding” and recursive self-improvement, began to be rolled out last year.
asserted
AI → begin → vibecoding
Those are very dangerous capacities, as I will explore below, and my complacency had been mostly based on the fact that those dangerous capacities didn’t exist.
asserted
capacities → explore → fact
And I am not the only one who is alarmed, as Coxon’s resignation and the push to slow AI development have made clear.
asserted
resignation → alarm → development
It’s time to take AI safety seriously.
asserted
It → ’ → safety
It’s reasonable to ask how AI might kill us all.
uncertain
AI → ’ → us
We can answer that question on several levels.
asserted
We → answer → levels
At the first level, AI can kill humans in the same way that humans kill humans.
asserted
humans → kill → humans
AI is already being used in lethal ways: Ukraine’s preferred weapon in the war with Russia isn’t tanks or guns, but small, unmanned drones.
asserted
weapon → use → Russia
These drones are being increasingly piloted entirely by AI.
asserted
drones → pilot → AI
It is therefore not hard to imagine a future where it is not just Russian soldiers being hunted and killed by swarms of AI-run killbots, but everyone.
asserted
soldiers → imagine → killbots
Another kind of risk comes from bioterrorism.
asserted
kind → come → bioterrorism
We already have the capability to design novel viruses or other biological agents.
asserted
We → have → viruses
That capability will only increase in the coming years, as AI opens up new frontiers of understanding and control over human biology.
asserted
AI → increase → biology
The blogger Noah Smith paints a picture of a near future where a misanthrope uses a cracked version of an AI to design a supervirus that spreads quickly, with a high mortality rate.
asserted
that → paint → rate
Biological risks like these seem particularly worrisome, since AIs can use them without harming themselves.
asserted
AIs → seem → themselves
Viruses already kill people, and new viruses can be particularly lethal as we saw with COVID-19; this kind of risk already exists.
asserted
kind → kill → risk
Again, the threat here is that AI may be capable of increasing this risk at much greater scale and efficiency.
uncertain
AI → increase → scale
At a second level, we should remember that AI is native to computer systems; that’s where it lives.
asserted
it → remember → systems
…and 107 more, not listed.
The Aporia
