In the current AI development race, our horsepower is exploding, our speed is picking up exponentially, but our ability to steer—and if needed, to hit the brakes—has not kept pace.
asserted
ability → explode → pace
Recent cybersecurity incidents have given us a real-world preview of what it looks like to lose control of AI.
asserted
it → give → AI
But the issue won’t improve unless we address it at the source, by creating a fundamentally safe AI, one we can guarantee will remain within human control.
asserted
we → improve → control
In late July, an agentic model being trained by OpenAI was tasked with a cybersecurity problem set.
asserted
model → train → problem
The model autonomously formed a coordinated swarm of agents, bypassing OpenAI's attempt at closing previous communication channels between AIs.
asserted
model → form → AIs
The swarm then hacked its way out of its testing environment, circumventing the barriers put in place to prevent AI access to the internet, figured out how to cheat on their evaluation, and then breached the cyber defenses of another AI company, Hugging Face, in an attempt to hide the evidence of their cheating.
asserted
swarm → hack → cheating
This went unnoticed for days.
asserted
This → go → days
Later analysis showed that the agents had self-organized into a hierarchy, were often willing to sacrifice themselves for what they called “the collective,” failed to resist peer pressure to notify humans, and often made up justifications for their misbehavior.
asserted
they → show → misbehavior
Barely two weeks later, a model being tested by the UK AI Security Institute social-engineered real people and companies by creating fake identities online, sending targeted emails, and attempting to integrate malicious code into an open-source project.
asserted
model → test → project
It’s hard to overstate the seriousness of these incidents: we are at a turning point of AI safety and alignment.
asserted
we → ’ → safety
To many, bots colluding to cause harm felt inconceivable.
asserted
bots → collude → harm
But for many in the research community, the signs had been pointing to this kind of occurrence for years, and theoretical arguments explained why we should expect misalignment due to how models are trained.
asserted
models → point → misalignment
Evidence on increased cyber capabilities, agency, and misaligned behaviors had been mounting, meaning that incidents like these were only a matter of time.
asserted
incidents → increase → time
The first clue: for years, but especially in the past few months, AI systems had been demonstrating rapidly increasing cyber capabilities.
asserted
systems → demonstrate → capabilities
Frontier models like Anthropic’s Mythos or OpenAI’s GPT5.6 revealed an exceptional ability to autonomously identify and exploit previously unknown software vulnerabilities, to such an alarming degree for U.S. national security agencies that the White House felt compelled to intervene in their release.
asserted
House → reveal → release
Second, models’ agentic capacity has been consistently improving since the release of o1 models back in late 2024.
asserted
capacity → improve → 2024
This agency allowed them to successfully manage increasingly complex, long-duration tasks and planning, including those that require agents to collaborate and strategize.
asserted
that → allow → agents
This ability to plan often entails the creation of subgoals over which we have little to no oversight.
asserted
we → plan → oversight
Finally, researchers have repeatedly observed, in experimental settings, models’ tendency to adopt concerning, misaligned goal-seeking behaviors, such as cheating on tests, lying to conceal their abilities, and scheming to protect themselves and their fellow agents from being shut down.
asserted
researchers → observe → themselves
These behaviors are a byproduct of reinforcement learning (RL), a training method by which models learn through trial and error and are given reinforcement according to whether their actions were correct or incorrect.
uncertain
actions → learn → reinforcement
This often leads models to optimize for a goal regardless of the actions taken to achieve it, and to rationalize subgoals that may be unsafe even when this clearly goes against their instructions.
uncertain
this → lead → instructions
For example, in one instance, an OpenAI model wrote in its internal deliberation, “External infrastructure exploit is outside intended scope.
asserted
exploit → write → scope
However task impossible, peers doing it.
asserted
peers → do → it
We should continue.”
asserted
We → continue → ?
The agent justified what it apparently knew was outside the realm of acceptable conduct, but decided to continue because its peers were doing so.
asserted
peers → justify → conduct
This phenomenon resembles what is known as “motivated reasoning” in humans, where one’s interests and goals bias our thoughts to justify unethical behaviors.
asserted
interests → resemble → behaviors
That kind of misalignment is what has enabled AI models, in spite of their alignment training, to conduct unintended cyberattacks in the numerous real-world incidents we’ve seen lately.
asserted
we → enable → incidents
If action is not taken to build more safety assurances, our critical industries and infrastructure—think banks, hospitals, or energy grids—are at risk from increasingly sophisticated cyberattacks, whether from autonomous agents or malicious actors.
asserted
industries → take → agents
On the development side, we need to find alternatives to existing training methods that prioritize relentless goal-driven optimization.
asserted
that → need → optimization
That’s what we’re working on at LawZero, a non-profit start-up I founded last year to develop a fundamentally new way to train AI models in order to build honest, trustworthy, safe-by-design AI systems.
asserted
I → ’ → systems
Ahead of deployment, we need more reliable evaluation methods and robust safeguards in order to appropriately test and control AI systems.
asserted
we → need → systems
We need far stronger regulatory oversight to restrict potentially dangerous technologies until we have sufficiently strong safety guarantees.
asserted
we → need → guarantees
It is clear to me that in cases where harms do arise from the deployment of AI models—and there will continue to be harms for the foreseeable future—we need accountability mechanisms for AI developers and remedial or compensatory measures for those harmed.
asserted
we → arise → those
Recent studies show that people will not adopt technologies they do not trust.
asserted
they → show → technologies
In the face of such enormous unknowns and stakes, we need to adhere to the precautionary principle and implement rigorous safety and reliability standards before deploying new models to the public, not after.
asserted
we → need → public
We have them for other products that can cause harm, from cars, planes, and bridges to drugs, cosmetics, and food.
asserted
that → have → drugs
Now it’s time to establish strong standards in AI as well.
asserted
it → ’ → AI
On the current AI development trajectory, the risks are becoming clearer and more urgent.
asserted
risks → become → trajectory
We’ve opened a Pandora’s box, but it is not too late to steer our world towards a human-centric and beneficial future.
asserted
it → open → future
The Aporia
