Earlier this month, frustrated by some of your social media posts, I challenged you to a public debate on AI.
asserted
I → frustrate → AI
I hadn’t realized that debate offers are to New Media what chum is to sharks.
asserted
chum → realize → sharks
My inbox was deluged by podcasters, debating societies, and general influencers, all offering to host.
asserted
all → deluge → podcasters
When I objected that maybe we should wait for you to accept before figuring out details, they all said great, that’s absolutely right, they would contact you right away.
asserted
they → object → you
Their people would reach out to your people.
asserted
people → reach → people
Their nephew knew your friend’s babysitter and they were confident they could relay the message.
uncertain
they → know → message
They had already rented an empty lot near your house where they would place a speaker blaring “RESPOND TO SCOTT ALEXANDER’S DEBATE OFFER!” at max volume, day and night.
asserted
they → rent → volume
I protested that probably asking you directly on Twitter was enough to get your attention, and maybe now we should give you some time to make up your mind before pressing further, but they said that no, sorry, they’d already hired the team from Inception to place subtle debate-related messages in your dreams.
asserted
they → protest → dreams
So I feel a little bad for unintentionally harassing you, and doubly so after you responded in your trademark reasonable and charitable style with An Open Letter To Scott Alexander.
asserted
you → feel → Alexander
You write that in-person debates seem confrontational and not truth-seeking.
asserted
debates → write → person
Apologizing for any inadvertent offense you may have given with your social media posts, you nevertheless deny my accusations of personal attacks, saying that your primary contribution to the discourse has been reasonable arguments about why AI might not be so dangerous after all (which you go on to list).
uncertain
you → apologize → which
You invite me to respond to these arguments in the same spirit of respectful mutual intellectual inquiry with which they are offered.
asserted
they → invite → which
I agree that in-person debates are confrontational and bad for truth-seeking.
asserted
debates → agree → seeking
But I didn’t propose a debate in order to seek truth.
asserted
I → propose → truth
I proposed it because, under California Penal Code § 415(1), it’s illegal for me to challenge you to a duel.
asserted
me → propose → duel
I think your public writing on this topic has been dishonorable.
asserted
writing → think → topic
Out of obligation, I will respond to the meaty arguments that you have set out for me.
asserted
you → respond → me
But what would be viscerally satisfying would be to make you get up on a stage where I read your own words to you in real time and ask “Really? Really?” after each sentence.
asserted
I → make → sentence
Then I could watch you squirm as you try to square your output with your status as one of America’s top public intellectuals.
uncertain
you → watch → intellectuals
So I’ll start by addressing the meaty arguments, move on to directly impugning your honor, and then we’ll decide whether to debate, duel, or seek some kind of synthesis of our conflicting views.
asserted
we → start → views
1: [Doomer scenarios have] an underdeveloped conception of intelligence which treated it as a quantity of power which may be extrapolated from animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience.
uncertain
latter → treat → omniscience
I argued that any intelligent system is a mechanism which is good at solving some problems but not others, and which is inherently limited by knowledge about the world attainable only by observation and experimentation.
asserted
which → argue → observation
You can split hairs, you can come up with some obscure sense in which that word is slightly off, but common-sensically it’s true.
asserted
it → split → which
At some point, this intelligence will exceed human intelligence.
asserted
intelligence → exceed → intelligence
My argument requires nothing more complicated or philosophically dubious than this.
asserted
argument → require → this
I have had a surprising amount of trouble conveying this argument to you.
asserted
I → have → you
At each GPT level, you seem to have believed it was implausible that AI would get more intelligent than it was already.
asserted
it → seem → level
Credit to you for starting early - you began predicting this in 2019, around GPT-2:
asserted
you → start → GPT-2
In 2022, you and I were part of a sprawling many-sided debate with Scott Aaronson and Gary Marcus.
asserted
you → sprawl → Aaronson
Marcus had noticed that GPT-2 and GPT-3 couldn’t solve basic math problems, like “I put two trophies on a table, then add another - the total number of trophies is now ______?”
asserted
number → notice → trophies
I argued that as AI scaled up, GPT-4, GPT-5 and so on would avoid these failure modes and become smarter.
asserted
GPT-4 → argue → modes
You, sticking to your claim that intelligence was not a meaningful lens through which to think about the problem, guessed that it wouldn’t:
uncertain
it → stick → problem
Of course, GPT-4 was able to add 2+1 just fine, GPT-5 could do harder problems still, and an internal OpenAI model just beyond GPT-6 recently solved Navier-Stokes.
uncertain
model → add → Stokes
In 2023, you tripled down on the claim, noting that the AIs of the day still couldn’t figure out that someone alive in the morning and evening was alive in the afternoon, and saying that “I doubt it will improve exponentially”:
uncertain
it → triple → afternoon
It is, in some cases, admirable to stick to one’s guns - but at this point I feel like you’re just being stubborn.
asserted
you → stick → point
I think we should acknowledge that each GPT generation will be better than the last, and that - while this will no doubt asymptote eventually - we have no compelling argument for placing the asymptote at any particular point, including “right here” or “just before the AI reaches human level”.
asserted
AI → think → level
Does this require reifying intelligence as a specific mysterious fluid?
asserted
this → require → fluid
You have ably written about how this seemingly absurd claim holds for humans - specifically, all intellectual abilities are correlated in a mysterious pattern called g, the basis of IQ tests.
uncertain
abilities → write → tests
But maybe this is only a quirk of humans, which cannot be extended to other minds?
asserted
which → extend → minds
Psychologists’ two favorite things are rats and IQ tests, so they have hardly failed to give IQ tests to rats.
asserted
they → fail → rats
…and 373 more, not listed.
The Aporia
