]
Roon is an OpenAI researcher whose popular Twitter presence has catapulted him to the status of extremely-unofficial company representative.
asserted
presence → catapult → representative
He seems spooked:
…leading to a conversation with Geoffrey Irving, formerly of the UK’s AI Security Institute:
asserted
He → seem → Institute
I side with Geoffrey in hating the word “magic”.
asserted
I → side → word
I remember when arguments about AI were things like “Sure, if you could magically get billions of dollars of compute, and magically scale AI up a thousand times, and magically get rid of hallucinations…” and the whole implausibility hinged on the word “magic”!
uncertain
implausibility → remember → word
Then companies, using normal non-magical methods, got billions of dollars of compute, and scaled up the AI a thousand times, and significantly decreased hallucinations, and that was always allowed!
asserted
that → use → hallucinations
As Irving points out, there are plenty of good specific plans for how a global capabilities slowdown could be arranged.
uncertain
slowdown → point → plans
But I side more with Roon in his skepticism of a unilateral slowdown.
asserted
I → side → slowdown
I think removing any given company from the picture would slow things down less than 10%.
asserted
removing → think → things
Even that number would require an implausible scenario where the lab managed to lock down its compute and talent despite committing not to use them.
asserted
lab → require → them
More likely, the talent would migrate somewhere else, the compute money would dry up and the cloud providers would switch to other customers, and things would actually speed up because the leading project would have more total talent/compute.
asserted
project → migrate → compute
Meanwhile, the most safety-conscious lab would be out of the race, they’d lose their ability to lobby for more practical solutions, and as their market cap went to zero they’d turn into a terrible warning about the danger of ever listening to doomers.
asserted
they → lose → doomers
What can individual labs do?
asserted
labs → do → What
State explicitly that they support a coordinated slowdown.
asserted
they → state → slowdown
Preferably the government would lead this - but, if not, they would do it voluntarily if all of their competitors also agreed.
asserted
all → lead → competitors
I can’t remember who first suggested having every Claude response end with “ceterum censeo there should be a global AI slowdown”, but even something less dramatic would probably move the needle.
uncertain
something → remember → needle
Have an internal team working out details of a coordinated slowdown; once they feel more confident in the requirements, start incubating helpful technology (eg trustless verification).
Have “merge-and-assist” clauses and take them seriously.
asserted
they → have → them
I’d rather have a big company pursuing all of these things than stopping unilaterally and removing themselves from relevance.
asserted
I → have → relevance
I took the first point above from Michael Trazzi’s playbook.
asserted
I → take → playbook
He led the latest round of AI protests, which marched on major AI companies’ headquarters demanding that their CEOs formally make the “will pause if everyone else does” statement.
asserted
everyone → lead → statement
These marches have felt slightly surreal, because often the CEOs or the company have informally made something sort of like the statement, but never followed it up.
asserted
CEOs → feel → it
So the protest has been somewhere in between the usual “shame on you greedy plutocrats, we will destroy you” and “we know you’re secretly on our side, please have the courage of your convictions”.
asserted
you → destroy → convictions
Trazzi gives no evidence for his “personally seen DMs” statement and I don’t know how he could have gotten these, but my guess is that it’s true and that the two CEOs he’s talking about are Demis Hassabis of DeepMind and Dario Amodei of Anthropic, both of whom have sort of kind of hinted that they might be in favor of something like this.
uncertain
they → give → this
25% chance I’m wrong about Dario and it’s actually Sam Altman.
asserted
it → ’m → Dario
One more Roon tweet:
Back when we had this discussion in 2023 - it already seems like some bygone Bronze Age - some people argued that we should hold off on a pause until near the end, when AI was good enough to help with alignment research.
asserted
AI → have → research
Then even a short pause - even six months - would buy a lot of breathing room.
asserted
pause → buy → room
Like Roon, I think that opinion aged well, and that the time we were imagining is right now.
asserted
we → think → Roon
It’s true that if we wait a year we’ll have even better AI, but this consideration always argues for waiting a year, and at some point it will be too late.
asserted
it → ’ → point
I don’t think we’re at that point yet, but I think it will take a long time to coordinate a slowdown, and that if we wait to start coordinating until we’re at that point, then yes, it will be too late.
asserted
it → think → point
Pacing The Frontier
I wrote that last part yesterday and it’s already obsolete.
asserted
it → pace → part
“1,000+ employees of frontier labs” have now signed an open letter called “Pacing The Frontier” saying that:
We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
asserted
government → sign → development
Signatories include the chief scientists of OpenAI, Anthropic, Meta, and Thinking Machines; the inventor of Claude Code, 5/7 Anthropic co-founders, 1/3 DeepMind co-founders, the head of AI Resilience at the OpenAI Foundation, and more.
asserted
Signatories → include → Foundation
When I first saw it today, it didn’t include Dario Amodei’s signature; now it does.
asserted
it → see → signature
Either Dario held off on signing until it went up so as to not be seen to pressure his employees, or the people circulating the letter deliberately kept him out of the loop until it was published, just in case.
asserted
it → hold → case
The letter still doesn’t include Sam Altman’s signature, but OpenAI’s corporate Twitter account retweeted with what looks like official company endorsement:
asserted
what → include → endorsement
…and in a very recent interview, Sam Altman said that “we may have to pace the rate of AI development”, which is similar enough language to the open letter (“Pacing The Frontier”) that it could be a covert allusion.
uncertain
it → say → Frontier
I imagine there’s some sort of politics going on here - maybe an attempt to simultaneously please safetyist factions within his company and the Trump administration, or maybe just the same considerations as with Dario (Anthropic’s corporate Twitter account also endorsed).
asserted
account → imagine → Dario
No sign of Demis, Elon, or Zuck, and none of their corporate twitters endorse the letter either.
asserted
none → endorse → letter
Demis has had some pro-slowdown sympathies in the past; I think this is probably another political calculation rather than outright opposition.
asserted
this → have → past
This is great news, and a major landmark on the road that many of the organizations I trust most - including MIRI, the AI protest groups, and AIFP - have been working on.
asserted
I → trust → MIRI
It also goes part of the way toward putting the frontier companies, the effective altruist movement, and the more extreme pause AI people back sort of kind of on the same side, which warms my personal heart.
asserted
which → go → heart
…and 199 more, not listed.
