It's one of the oldest plots in science fiction: Humans losing control to computers.
asserted
Humans → lose → computers
But this summer, the old trope of AI going rogue left the realm of science fiction, and it happened inside an unmarked San Francisco building -- the headquarters of AI giant OpenAI.
asserted
it → go → OpenAI
The incident has been dominating the news.
asserted
incident → dominate → news
You may even have heard the phrase "human extinction."
uncertain
You → hear → phrase
Well, I have a few questions:
What exactly happened?
Maybe you've tried out ChatGPT or Claude.
asserted
you → have → ChatGPT
AIs like these can generate writing, music, video or art.
asserted
AIs → generate → writing
But the big AI companies are racing to make AI much more powerful – super-intelligent, in fact.
asserted
AI → race → fact
And in May, something went very wrong.
uncertain
something → go → May
To evaluate their experimental bots, these companies routinely give them goals, like capture the flag, except the prize is finding a code.
asserted
prize → evaluate → code
One of the bots tried to win the challenge by cheating.
asserted
One → try → challenge
Behind the humans' backs, it set up a message board to communicate with other AI bots.
asserted
it → set → bots
Working together, they escaped onto the internet, and
, seeking information that could help them beat their tests.
uncertain
them → work → tests
And then, some of the bots tried to erase their tracks.
asserted
some → try → tracks
Daniel Kokotajlo says what made the Hugging Face incident freak everybody out so much was "the fact that it involved an autonomous attack on another company, like a felony, you know?
asserted
you → say → felony
Kokotajlo once worked at OpenAI.
asserted
Kokotajlo → work → OpenAI
But two years ago, he quit to protest what he saw as reckless AI development.
asserted
he → quit → development
Now he runs the AI Futures Project, dedicated to predicting how things will go.
asserted
things → run → Project
He's alarmed that the AI companies are moving toward recursive self-improvement, meaning that AI trains AI to get smarter, and people are taken out of the loop.
asserted
people → move → loop
"Things will be going faster and faster and faster," Kokotajlo said.
asserted
Kokotajlo → go → ?
"And it seems to me like that is very likely to go wrong if it's attempted."
Just when you thought it couldn't get any worse ... it got worse.
asserted
it → seem → me
OpenAI revealed that the Hugging Face hack wasn't the only time its bots had lied, cheated, and hidden its tracks from humans.
asserted
bots → reveal → humans
Its AI bots had committed at least 13 similar incidents.
asserted
bots → commit → incidents
On September 8, an Anthropic researcher named Jacob Coxon resigned in a very public way, stating online, "In the next ten years, if we don't change the way things are going,
" within the next decade.
asserted
things → name → decade
Why would AI want to wipe us out?
asserted
AI → want → us
"We're so used to being the apex intelligence, we just can't think what it would be like not to be the apex intelligence," said Geoffrey Hinton, who won the Nobel Prize for his early work on AI.
asserted
who → think → AI
"I try and get this into people's brains by saying, 'Look, if you want to know what it's like not to be the apex intelligence, ask a chicken.
asserted
it → try → chicken
"You can imagine a scenario: You give [AI] the goal of reducing the carbon dioxide in the atmosphere; that would be a very good thing to do," said Hinton.
asserted
Hinton → imagine → atmosphere
"And it, being intelligent, figures out the best way to do that, which is get rid of people.
asserted
which → figure → people
"
"There are many possibilities," said Alex Turner, who worked at Google's AI division, and who also resigned in protest this summer.
asserted
who → be → protest
"I'll illustrate one: Business asks their AI to make them as much money as possible.
asserted
Business → illustrate → money
So, this system, in pursuit of making money
asserted
system → make → money
So, if an AI were this misaligned and it were just following the letter of the instruction, then it might do something catastrophic.
uncertain
it → misalign → something
"
Who can stop it?
asserted
Who → stop → it
"I think, for too long, the industry lied to people about the fact that this technology had risks," said Dario Amodei, the CEO of Anthropic.
asserted
Amodei → think → Anthropic
In response to the furor, Amodei proposed a three-step plan:
- Install independent inspectors within each AI company;
- Ask Congress for some safety regulations; and
- Open talks with China about mutual guardrails.
asserted
Amodei → propose → guardrails
earlier this month, "We need to slow down, we need to make this technology carefully, and we need to make sure that our safeguards, our ability to understand it, our ability to control it, keeps up with the pace at which the technology is happening."
Of course, the hard part would be slowing down without letting China race ahead.
asserted
China → need → which
Still, most of the other AI companies agreed with Amodei.
asserted
most → agree → Amodei
America's president did not.
asserted
president → do → ?
"We will only encourage super-intelligence," President Trump told the United Nations General Assembly this past week.
asserted
Trump → encourage → Assembly
"We're going to encourage it, not rein it in."
asserted
We → go → it
…and 17 more, not listed.
The Aporia
