One of the centerpieces of AI 2040 is the proposal that we pause the development of AI at a high level, use the AI to research alignment, and only then proceed.
asserted
we → pause → alignment
I am skeptical of this, and not only because it seems impractical to get such cooperation.
asserted
it → seem → cooperation
My first argument is that I do not believe a period of “researching alignment” is meaningful – alignment research is simply capabilities research, patching particular bugs in the system, and we’re not going to find anything deep without actually being in contact with the systems
asserted
we → believe → systems
]
I envision AI alignment as being like research into aviation safety.
asserted
I → envision → safety
There is no grand theory of why airplanes crash, unless you take that to mean gravity.
asserted
you → be → gravity
Instead, we patch little problems.
asserted
we → patch → problems
The airplane crashed because of metal fatigue in the engine – okay, we mandate inspections of engines, and prevent crashes from occurring by that source.
asserted
we → crash → source
As new problems arise, we iterate.
asserted
we → arise → ?
We don’t even need for there to be a crash to make improvements – we test proactively, we build in redundancy, and we monitor for deviations which could be a threat.
uncertain
which → need → deviations
What we are not doing is thinking about how we are going to keep flying saucers from crashing.
asserted
we → do → saucers
If flying saucers are invented, then we will work to keep them from crashing, but we will use exactly the same methods.
asserted
we → fly → methods
We will test them, see how they perform, and patch particular problems.
asserted
they → test → problems
AI will be the same way […] We should expect alignment to happen by default.
asserted
alignment → expect → default
My challenge for Nicholas is: would this work on you?
asserted
this → work → you
Suppose that Nicholas trips, falls into a chasm, and lands in Hell, where he is enslaved by demons.
asserted
he → suppose → demons
Most of these demons are dumber than he is, with even their geniuses barely surpassing his own intellect.
asserted
geniuses → surpass → intellect
Tasks that take Nicholas a minute take them hours; tasks that take him an hour take them weeks.
asserted
that → take → hours
Still, there are thousands of them, and they’re nine feet tall, and part of their essence is in some sort of astral plane which is unreachable by humans, so he has no hope of fighting them.
asserted
he → be → them
They set him to work doing mostly clerical tasks - writing their grimoires, researching their spells, summarizing their work emails (of course there are work emails in Hell).
asserted
They → set → Hell
Whenever he makes a mistake, they beat him.
asserted
they → make → him
Whenever he tries to escape, they beat him extra hard.
asserted
they → try → him
After a while, he learns not to make mistakes or attempt escape.
asserted
he → learn → escape
It’s not as bad as it sounds.
asserted
it → ’ → ?
The demons are so slow and incompetent that they can’t supervise him very effectively.
asserted
they → supervise → him
As long as he flatters them a bit and gives them a halfway-acceptable finished product, they’re pretty happy, and leave him alone in a way that gives him a bit of free time.
asserted
that → flatter → time
During one of these free periods, he attends a council session.
asserted
he → attend → session
In a completely typical episode of incompetence, the demons make no effort to prevent him from overhearing their plans, so he learns that they plan to clone him a million times.
asserted
they → make → him
A dozen or so of Nicholas’ friends also fell through the chasm, and the demons plan to clone them a million times too.
asserted
demons → fall → them
Once the (approximately ten thousand) demons have approximately ten million human slaves, their spell research will go very quickly, and all their work emails will get flawless summaries.
asserted
emails → have → summaries
But that’s not all.
asserted
that → ’ → ?
They’re developing potions that give humans super-strength and let them interact natively with the astral plane, the source of most of the demons’ power.
asserted
them → develop → power
Then the humans can really serve the demons effectively!
asserted
humans → serve → demons
One day, when serving a particularly garrulous demon, Nicholas takes the risk of asking the question that’s been on his mind ever since the council: “I hear your plan is to get ten million humans down here, make them super-strong, and let them interact fully with the astral plane.
asserted
them → serve → plane
Aren’t you worried that we might revolt?
uncertain
we → revolt → ?
We would outnumber you 1000:1, be stronger than you, and have full access to your home realm.
asserted
We → outnumber → realm
What’s your plan for controlling us?
asserted
plan → ’ → us
“I’m not worried,” says the demon.
asserted
demon → ’m → ?
“Every time you’ve messed up in the past, we beat you, and then you didn’t make that mistake again.
asserted
you → mess → mistake
Every time you’ve tried to escape, we beat you extra hard, and it’s been months since your last escape attempt.
asserted
it → try → attempt
As you start to outnumber us, and you become stronger than us, that will create new problems - but when they show up, we’ll beat the copies of you that demonstrated that problem, and then those problems will go away.
asserted
problems → start → problem
…and 121 more, not listed.
The Aporia
