The Story So Far
Mechanistic interpretability is the science of “reading an AI’s mind”.
asserted
interpretability → read → mind
Large language models are “grown, not built”.
asserted
models → grow → ?
Researchers run training data through a neural network.
asserted
Researchers → run → network
Eventually this creates a working AI; nobody really knows how.
asserted
nobody → create → AI
But a neural network is just a set of simulated neurons on a computer.
asserted
network → simulate → computer
The person with the computer can see the neurons, the connections between them, and which ones activate when the AI answers questions.
asserted
AI → see → questions
So it seems like it should be possible to “reverse engineer” the AI.
asserted
it → seem → AI
This would be scientifically useful: understand how AIs work, with possible relevance to human cognition.
asserted
AIs → understand → cognition
It could also be practically useful: find the circuits responsible for dishonesty, hallucination, bias, and other negative behaviors, then redesign those circuits.
uncertain
It → find → circuits
A modern AI has millions of neurons and trillions of connections between them (aka “parameters”, “weights”).
asserted
AI → have → them
Their structure is apparently nonsensical: researchers started by seeking a 1:1 mapping between neurons and concepts, like a neuron that always fired when the AI was thinking about cats, but quickly learned that nothing like that existed.
asserted
nothing → start → that
Their first big breakthrough came in 2023.
asserted
breakthrough → come → 2023
Rather than a 1-to-1 neuron-to-concept mapping, they discovered a many-to-many mapping.
asserted
they → discover → mapping
Neurons 1 and 2 firing together might mean cat; 1 and 3 firing together might mean chair; 2 and 3 firing together might mean bread.
uncertain
2 → fire → bread
Such mappings let AIs with “only” tens of thousands of neurons represent millions of concepts.
asserted
AIs → let → concepts
There were far too many of these combinations for humans to map, but researchers set other, smaller AIs to mapping them and met with some early success - like this functional combination of neurons (“feature”) representing “God” in an artificially-simple 512-neuron AI:
Spirits were high.
asserted
Spirits → be → AI
Scale up the AI interpreter, and maybe we could do anything!
uncertain
we → scale → anything
Philosophers could discover the fundamentals of cognition.
uncertain
Philosophers → discover → cognition
Doomers could solve alignment by tracing the circuits involved in moral reasoning.
uncertain
Doomers → solve → reasoning
Tens of millions of dollars poured into the field.
asserted
millions → pour → field
Superstar interpretability researcher Chris Olah got an audience with the Pope (okay, fine, this was mostly unrelated).
asserted
this → get → Pope
By 2024-2025, the clarity of many-to-many mapping had dissolved, and everyone was back to feeling confused and miserable.
asserted
everyone → dissolve → mapping
Strategies that had successfully analyzed artificially-simple research-subject AIs failed to work on real language models.
asserted
that → analyze → models
Different teams making many-to-many maps of the same AI reported different results.
asserted
teams → make → results
And humans watching supposedly-mapped AIs more closely found that seemingly-crisp features were much vaguer and worse than previous analysis had led them to believe: a feature which they thought meant “God” might activate on any vaguely religious or awe-inspiring text, or even text with no logical connection whatsoever.
uncertain
God → watch → connection
As for attempts to use the many-to-many maps for useful work - like AI lie detection - they generally failed to outperform simpler methods.
asserted
they → use → methods
So researchers abandoned the hope of One True Technique - even if that technique was “actually understanding things” - and went back to the drawing board.
asserted
technique → abandon → board
Now mechanistic interpretability is starting to feel exciting again - but also more intimidating than ever, as one tries to keep all the jargon straight.
asserted
one → start → jargon
This list is my log and personal reference document for remembering what everything is.
asserted
everything → remember → ?
It’s loosely inspired by the concept breakdown in Benjamin Sturgeon’s Contributing To Technical Safety Research In The AI Safety End Game, which itself follows the Claude Mythos Preview System Card.
asserted
which → inspire → Card
Imagine an AI with three neurons: [0, 0, 0]
The researcher makes the AI think of cats in various contexts, averages them all out, and gets some set of activations like [0.8, 0, 0.35].
asserted
AI → imagine → ]
Imagine the AI as operating in a Cartesian space where each of its neurons is one dimension.
asserted
each → imagine → neurons
The vector from [0, 0, 0] to [0.8, 0, 0.35] defines a direction in that space.
asserted
vector → define → space
In a linear probe, researchers compare the true direction of the AI’s thoughts to one of these pre-defined concept-representing direction vectors.
asserted
researchers → compare → vectors
Because activation-space has thousands of dimensions, its behavior is counterintuitive, and a particular thought can be in thousands of “directions” at once and yet still “closer” to some directions than others.
asserted
thought → have → others
Experiments confirm that when an AI’s thoughts are close to the cat direction, this is a good sign that it’s thinking about cats.
asserted
it → confirm → cats
Why doesn’t this solve the whole problem?
asserted
this → solve → problem
First, this teaches us very little about how AIs work - how, for example, they might answer a difficult query in veterinary medicine.
uncertain
they → teach → medicine
At best, it’s a primitive lie detector: if an AI claims it isn’t thinking about cats, you can check its work (either by probing the concept of cat, or by probing the concept of lying!)
uncertain
you → ’ → lying
Second, it’s only as good as the original linear probe.
asserted
it → ’ → probe
…and 165 more, not listed.
The Aporia
