Last month, some of Silicon Valley’s biggest companies signed an open letter supporting open-weights AI.
asserted
some → sign → AI
Open weights AI is like open-source software, where the creator makes the raw code publicly available for free download.
asserted
code → make → download
It’s good insofar as it’s the only way an AI can truly be the user’s property, as opposed to something that companies like OpenAI or Anthropic temporarily let you use subject to their corporate guidelines and increasingly-nanny-state-like restrictions.
asserted
you → ’ → guidelines
If AI becomes the linchpin of the future, open weights AI feels like the sort of thing that could be the difference between being free yeomen vs. corporate serfs.
uncertain
that → become → serfs
It’s bad insofar as it removes the possibility of gatekeeping and lets criminals commit crimes with it.
asserted
criminals → ’ → it
Open weights AI could be used for hacking, child pornography, harassment, or terrorism (the weights can’t commit the terrorism themselves, but they could give bomb-making or bioweapon-making advice).
uncertain
they → use → advice
Since AIs have gotten very good - maybe superhuman - at hacking lately, the specter of a world where anyone can hack any site has gotten people grumbling that maybe open weights should be banned.
asserted
weights → get → site
It doesn’t help that China produces the best open weights AI, making the idea seem foreign and almost unpatriotic.
asserted
idea → help → AI
Proponents counter that “when AI is outlawed, only outlaws will have AI”, arguing that bad people will get open weights AI regardless, and good people can use open weights AI to defend themselves.
asserted
people → counter → themselves
With the recent open letter, companies including Microsoft, NVIDIA, OpenAI, Intel, Amazon, Meta, Hugging Face, and over a hundred others have come out in favor of this position.
asserted
companies → include → position
Who’s leading the other side?
asserted
Who → lead → side
Nobody’s admitted to it.
asserted
Nobody → admit → it
Some parts of the Trump administration lean anti-open-weights on China hawk grounds, but have stopped short of explicitly asking for a full ban.
asserted
parts → lean → ban
Anthropic, the most notable omission on the pro-open-weights letter, made an ambiguous statement supporting “open-weights models that don’t have dangerous capabilities” - but the industry expects open weights models to have dangerous hacking capabilities within a year, and AFAICT the letter didn’t address that beyond inviting readers to draw the obvious conclusion.
asserted
letter → make → conclusion
In the absence of a more obvious opponent, some open weights supporters suspect our conspiracy - the loose band of AI safety advocates, effective altruists, rationalists, and pause activists who worry about existential risk from superintelligence.
asserted
who → suspect → superintelligence
By design, open weights AI is outside centralized control, and so impossible to permanently align against either human misuse (eg terrorism) or loss of control (eg AI turning against humans).
asserted
AI → align → humans
Even if its creator trains it not to hack, anybody in the world can download the weights and retrain the AI to hack all day long.
asserted
anybody → train → AI
But in fact, most AI safety organizations have remained quietly neutral, and I don’t know of any who make this a centerpiece of their activism1 (though I’m not 100% up-to-date on the whole landscape; if you know of one, tell me).
asserted
you → remain → me
A few have proposed policies that are contingently incompatible with open weights AI existing, but they all frame it as collateral damage rather than something they’re excited about eliminating.
asserted
they → propose → damage
I’m also neutral about open weights AI.
asserted
I → ’m → AI
I think it probably won’t be long-term sustainable, but I’m happy to wait for this to become clear in the normal course of things rather than expend effort and political capital to ban it immediately.
asserted
this → think → it
Currently nobody knows how to align AI, so it’s not like the big companies have things under control and the open weights hobbyists are going to ruin it for everyone.
asserted
hobbyists → know → everyone
But even if the big companies did get things under control, the takeover threat from open weights would be limited.
asserted
threat → get → weights
The closed source frontier is ~6 months ahead of the best open weights model; this has remained true for several years and seems likely to remain true in the future.
asserted
this → remain → future
If closed weights AI is aligned, but open source dangerous, the closed weight AIs will have six months to warn us, prepare for the danger, and chart a strategy.
asserted
AIs → align → strategy
Even afterward, the offense-defense balance will lean in our favor.
asserted
balance → lean → favor
AIs have already displayed the ability to hack effectively.
asserted
AIs → display → ability
And you can tell how worried Anthropic is about bioterrorism by how quickly Claude Fable seizes up when you ask it a biology question (the example below is obsolete; it’s slightly more graceful than this now):
asserted
it → tell → this
But 9-11, COVID, and the Hugging Face incident all suggest a similar theory of political change: the body politic hates preparing for impending threats, but loves reacting (some would say over-reacting) to them after they happen.
asserted
they → suggest → them
Ask people to bear the slightest cost in preparing for an approaching disaster, and they’ll call you a dirty fascist tyrant; urge the slightest restraint after the first foreshock of the disaster hits, and they’ll call you a weak unpatriotic anarchist.
asserted
they → ask → you
Solve for the equilibrium, and the thankless and political-capital-guzzling route of urging preemptive action should be taken only when waiting until the first foreshock would be too late.
asserted
foreshock → solve → action
The risk of superintelligent AI takeover passes this test.
asserted
risk → pass → test
Like other smart adversaries - for example, the Imperial Japanese at Pearl Harbor - AI will try its hardest to avoid alerting its intended victims until it thinks that it’s fully prepared and can execute a sudden decapitation strike.
asserted
it → try → strike
Unlike the Imperial Japanese, a superintelligence will be smart enough not to bungle the calculation.
asserted
superintelligence → bungle → calculation
Sitting around waiting for it to show its hand would be folly, not to mention that it could take years of alignment research to be ready for the threat.
uncertain
it → sit → threat
But the risk of criminals using open weights AI to hack people doesn’t pass the test.
asserted
criminals → use → test
Fine, so criminals use open weights AI to hack people.
asserted
criminals → use → people
RIP them, but hundreds of people get hacked every day.
asserted
hundreds → rip → people
There will be some number of billions of dollars in damage, some tech companies with silly names will get sued for allowing security breaches, and then everyone will panic and ban open-weights AI.
asserted
everyone → sue → AI
Or who knows, maybe the “good guy with an AI” people are right and this won’t happen and some other Chinese AI will be able to protect them.
asserted
AI → know → them
…and 22 more, not listed.
The Aporia
