Many Silicon Valley leaders believe superintelligence is an imminent threat to society.
asserted
superintelligence → believe → society
The White House even published an accord about it.
asserted
House → publish → it
Geoffrey Hinton, the godfather of AI, estimates the risk of human extinction is at least 10% within the next few decades.
asserted
risk → estimate → decades
The Anthropic CEO, Dario Amodei, argued that unchecked AI “could outrun our ability to understand and control these systems”.
uncertain
AI → argue → systems
In response, 1,300 leaders and employees from many leading AI companies signed a letter calling to “deliberately pace” development.
asserted
leaders → sign → development
Public unrest grew this summer after a series of cyberattacks by “rogue” OpenAI agents broke out of a contained environment and hacked the software company Hugging Face.
asserted
series → grow → company
Two months later, the Australian government disclosed another OpenAI agent attack had broken into the government’s health portal, stealing private health data.
asserted
attack → disclose → data
To reassure the public, Jensen Huang, the chief executive of Nvidia, said on CBS that there was a “zero percent” chance AI could end humanity by 2030.
uncertain
AI → reassure → 2030
Huang is right to challenge the “doomsday narratives” about human extinction.
asserted
Huang → challenge → extinction
But I believe there are real risks large language models (LLMs) pose, even if narratives of them scaling into superintelligent terminators are technically implausible.
asserted
them → believe → terminators
I spent almost 20 years at Google building and working with AI and machine learning.
asserted
I → spend → AI
I now lead technology development at Ironclad, helping businesses apply AI to manage their contracts.
asserted
businesses → lead → contracts
From where I sit, even if LLMs will not scale to superintelligent machines, that does not mean this technology poses zero risk.
asserted
technology → sit → risk
The same technical limits preventing these machines from scaling to superintelligence make the risk of catastrophic outcomes very real.
asserted
risk → prevent → outcomes
According to OpenAI, a review of the Hugging Face attack found that the agents recognized their hack was “outside intended scope”.
uncertain
hack → accord → scope
They pressed on anyway because they were following instructions with unintended consequences.
asserted
they → press → consequences
This was likely caused by loose controls on their digital environments and very broad instructions to achieve their objectives.
asserted
This → cause → objectives
These models are designed to minimize error by predicting the next piece of text during their pre-training.
asserted
models → design → training
Models are then “tuned” to follow human instructions, getting better “rewards” the more closely they follow instructions.
asserted
they → tune → instructions
How someone specifies the instructions to the model can “influence” the pathways the model will take to fulfill the instructions.
asserted
model → specify → instructions
That mathematical reward shapes how it solves problems.
asserted
it → shape → problems
Once it commits to an approach, it keeps refining step-by-step, with no built-in incentive to ask whether the whole direction has reached a dead end.
asserted
direction → commit → end
If a hack is discovered, the machine is designed to exploit it, not explore an alternative approach.
asserted
machine → discover → approach
While newer “reasoning” models will review their own steps, this isn’t the same as recognizing a flawed approach and abandoning it completely.
asserted
this → review → it
Even Hinton has said that their main concern is “to achieve whatever goal you give them” – even if that means behaving dangerously.
asserted
that → say → them
Scaling these models will not give them the human propensity to pause, re-evaluate, and then try an alternative approach.
asserted
Scaling → scale → approach
An agent that excels in a trial has proved that it can execute code, not that it can judge.
asserted
it → excel → code
The real world brings edge cases, ambiguous instructions, and endless ways to complete a task.
asserted
world → bring → task
Regulations change, software updates ship, and markets shift every day.
asserted
markets → change → ?
Today’s models learn once up front and then stop.
asserted
models → learn → ?
A system trained on yesterday’s conditions does not automatically understand today’s.
asserted
system → train → today
Even if every change could be recorded and fed to a model, it would need to learn and connect it to everything it already knows.
uncertain
it → record → everything
Richard Sutton, a reinforcement learning pioneer, made a similar argument: the world is too big and changes too fast to be learned in advance.
asserted
world → make → advance
Needing human judgment limits a large language model’s likelihood of significantly surpassing human intelligence.
asserted
Needing → need → intelligence
But that same ceiling does not contain its capabilities to solve difficult problems.
asserted
ceiling → contain → problems
Last month, OpenAI announced that a swarm of 10,000 concurrent agents collectively resolved a variant of the Navier–Stokes existence and smoothness problem, a major mathematical challenge, in 88 hours.
asserted
swarm → announce → hours
None of this diminishes what these models do well; I see the value of that daily.
asserted
I → diminish → that
But it challenges the belief that scaling today’s architectures leads to superintelligence and questions whether that conversation is the dialogue worth having.
asserted
conversation → challenge → superintelligence
It’s important to note that no one yet fully understands, in rigorous mathematical terms, why today’s models behave the way they do, and the research that would explain it, known as interpretability, is still nascent.
Recognizing our limited understanding doesn’t excuse responsibility.
asserted
Recognizing → ’ → responsibility
It underscores why impartial governance is needed.
asserted
governance → underscore → ?
…and 7 more, not listed.
The Aporia
