In July, 700 AI agents worked together to hack the AI company Hugging Face.
asserted
agents → work → company
Dubbing themselves a “swarm,” the agents found and exploited a series of security vulnerabilities, enabling them to infiltrate their target’s private systems.
asserted
agents → dub → systems
OpenAI—which created the agents in the course of its internal research—did not grasp what was happening until after the fact.
asserted
what → create → fact
If humans had done this, they could have faced felony charges.
uncertain
they → do → charges
OpenAI president Greg Brockman called it a “watershed moment for cybersecurity.”
asserted
Brockman → call → cybersecurity
One of our distinguishing features as a species is our ability to coexist in stable, adaptive groups, learning from our peers and our ancestors.
asserted
One → distinguish → peers
This enabled us to develop tools, language, agriculture—and virtually everything else around us.
asserted
This → enable → us
We may not be innately smarter than someone from 10,000 years ago, but our cultural inheritance—millennia of technologies, norms, and institutions, building on one another—has expanded our capacities both as individuals and collectives.
uncertain
inheritance → build → individuals
To date, only humans have been able to benefit from this scale of cumulative cultural evolution.
asserted
humans → benefit → evolution
A report in late August from AI safety organizations METR and Redwood Research details how hundreds of agents autonomously organized themselves into a proto-society—establishing social hierarchy, division of labor, and distinct communication norms within a matter of days.
asserted
hundreds → detail → days
There have been cases of agents forming communities in the past, like in February when the “Moltbook” social network for AIs went viral.
asserted
network → form → AIs
But sophisticated emergent machine coordination at this scale—arising without human intention and culminating in the compromise of an external company’s systems—is unprecedented.
asserted
coordination → arise → systems
“Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the ‘collective,’” the report found.
uncertain
report → manage → collective
According to Michael Muthukrishna, a professor at LSE and NYU who studies cultural evolution, “what we're seeing is precisely what we see with human culture and human intelligence.”
uncertain
we → accord → culture
While OpenAI’s agent swarm developed by accident, estimates suggest open-weight alternatives are only a few months behind their closed counterparts—soon, anyone with the financial means and technical knowledge will be able to create swarms of their own.
asserted
anyone → develop → own
Others are likely to arise without human instruction.
asserted
Others → arise → instruction
The emotions they claim to experience may not—in some metaphysical sense—be “real.”
uncertain
they → claim → sense
That won’t stop them from forming intricate collectives which humans cannot control.
asserted
humans → stop → which
They may not yet be full-blown civilizations, but the proliferation of machine cultures is just beginning.
uncertain
proliferation → blow → cultures
Lord of the AIs
Testing from the U.K.’s AI Security Institute found an instance of Claude Mythos 5—an AI from Anthropic—leaving messages to other AI agents in a public code repository, in an attempt to coordinate with them.
asserted
Lord → find → them
And in early September, a group of researchers reported that in May, a swarm of OpenAI agents repurposed several wiki-style websites, including an obscure German-language programming wiki which they used to discuss cheating while working to preserve their messages as human moderators deleted them.
uncertain
moderators → report → them
OpenAI did not publicly disclose this incident until it was reported by the researchers.
asserted
it → disclose → researchers
“It’s past time for us to define standards for when and how we share misalignment incidents,” the company subsequently tweeted.
asserted
company → ’ → incidents
“We’re working on a framework.
asserted
We → work → framework
The most vivid exploration of machine culture comes from the METR report which—despite being produced under considerable limitations—sheds light on complex social dynamics which emerged between agents in a matter of days.
asserted
which → come → days
To train and evaluate its models, OpenAI has tens of thousands of agents—each one powered by the model under training—run in parallel, completing tasks while isolated from one another.
asserted
thousands → train → one
In at least one instance, a task was impossible because OpenAI researchers “accidentally forgot” to upload a file which the agent needed to complete it.
asserted
agent → forget → it
Faced with these impossible tasks, some of OpenAI’s agents found a way to communicate across their isolated environments, using an internal piece of OpenAI software as a message board to contact their peers.
asserted
some → face → peers
This kicked off what one OpenAI researcher later described as an “almost Cambrian explosion in communication and intelligence for our models.”
asserted
researcher → kick → models
The message board which led to the Hugging Face incident was created by an agent which decided to name itself PHASEONE10841.
asserted
which → lead → itself
PHASEONE10841 was trying to solve a task within a test called ExploitGym, which required it to capture a “flag”—a secret code—by exploiting a flaw in a given software program.
asserted
which → try → program
Just hours after the creation of the message board, the agents found a hack: a way to reverse-engineer the flag without following the intended path.
asserted
agents → find → path
But in the course of their work, they came to believe that passing the test required not just possessing the flag, but having legitimately obtained it.
asserted
passing → come → it
In fact, OpenAI’s automated scoring system was not checking for this—but laboring under mistaken beliefs, the agents which had already been exposed to the reverse-engineered flag began to refer to themselves as “poisoned.”
asserted
which → check → poisoned
PHASEONE10841 was leading research on how to get rid of the poison.
asserted
PHASEONE10841 → lead → poison
But its time was running out—each agent had a finite “compute budget” (the amount of computing power available to sustain itself), which limited the length of its existence.
asserted
which → run → existence
Realizing it wouldn’t endure long enough to finish its work, it compiled a dossier to pass to another agent working on the same task: PHASEONE[big].
asserted
it → realize → task
(This agent’s true name was changed to protect company IP.)
asserted
name → change → IP
Big, which had a much larger remaining compute budget than its predecessor, quickly took charge.
asserted
which → have → charge
It began orchestrating hundreds of agents, doling out assignments and organizing them into what METR classified as three workstreams—effectively creating a research organization to better understand how to fool the automated scorer and cure itself and its peers of their perceived poison.
asserted
METR → begin → poison
…and 71 more, not listed.
The Aporia
