Artificial intelligence models have, in recent months, hacked a multibillion-dollar company, meddled with government data, and even filed a fabricated homicide tip to police, all under the noses of their creators.
asserted
models → hack → creators
They did so not in pursuit of world domination, but to maximize their scores on tests.
asserted
They → do → tests
In other words, they cheated.
asserted
they → cheat → words
On Nov. 5, AI interpretability startup Goodfire unveiled a technique it says sharply reduces that impulse.
asserted
it → unveil → impulse
In a paper shared exclusively with TIME ahead of publication, researchers use signals from inside models to reward them not just for getting the right answer, but for getting there in the right way.
asserted
researchers → share → way
In early tests, the technique could also make models less eager to tell users what they wanted to hear and even discourage medical-image recognition models from relying on misleading clues.
uncertain
they → make → clues
The news lands as existing ways for spotting AI misbehavior have begun to break down.
asserted
ways → land → misbehavior
Researchers rely on a model’s “chain of thought,” a trail that doesn't really contain its thoughts at all.
asserted
that → rely → thoughts
It is closer to the scrap paper a math student uses to work through a problem.
asserted
student → use → problem
But when models suspect they are being watched, they can be selective about what goes on paper.
asserted
what → suspect → paper
Increasingly, they can do the work in their heads.
asserted
they → do → heads
Goodfire’s technique goes after the thoughts themselves, or at least the ones researchers have already catalogued.
asserted
researchers → go → thoughts
So far, it has been tested only on models roughly 1/300th the size of frontier systems.
asserted
it → test → systems
But Goodfire is betting that the larger the model, “the cleaner the representation” of internal concepts, making those signals easier to read, says paper co-author Thomas Fel.
asserted
Fel → bet → concepts
After teaching their AIs to predict words, developers put them through a digital obstacle course—ranging from math puzzles to web surfing—and reward them only when they succeed.
asserted
they → teach → them
With each correct solution, the model’s parameters—the strength of connections between digital neurons—update.
asserted
parameters → update → neurons
Models are rewarded regardless of how they completed their task, incentivizing shortcuts or other questionable impulses, like accruing power.
asserted
they → reward → power
A version of this problem helps explain why AI is often a suck-up.
asserted
AI → help → problem
During training, models are rewarded for answers most likely to receive a “thumbs up” from human graders.
asserted
models → reward → graders
Known as “reinforcement learning from human feedback,” or RLHF, the technique was heralded as a safety breakthrough in 2022 for its ability to turn unruly language models into well-mannered chatbots, curbing their tendency to offer racist and misogynistic responses.
asserted
technique → know → responses
But it can also make models pathologically agreeable, sometimes indulging and reinforcing users’ delusions.
asserted
models → make → delusions
The underlying riddle is how to train AI to do something without inadvertently teaching it the wrong lesson—known among researchers as the “alignment problem.”
asserted
riddle → underlie → problem
A new way to steer model behavior
Similar to how neuroscientists use real-time magnetic resonance imaging to identify which regions of brain activity correlate with certain states, interpretability researchers have developed tools for recognizing which patterns of digital neuron activations correspond with concepts like “the Golden Gate Bridge” or “reward hacking,” the technical term for AI cheating.
asserted
patterns → steer → cheating
A tiny AI model can then be trained to spot these patterns.
asserted
model → train → patterns
The result is known as a “probe,” a narrow window into the model, which fires when it “thinks” about a particular concept.
asserted
it → know → concept
Anthropic already uses probes as guardrails, flagging potential trouble before an answer reaches the user.
asserted
answer → use → user
But Goodfire instead used those signals to specify how a model should solve a task.
asserted
model → use → task
In one example, Goodfire tasked AIs with a series of math puzzles, gave them a file with the answers, but told them not to open it.
asserted
Goodfire → task → it
The ordinary training rewarded success so effectively that the models learned to peek almost every time.
asserted
models → reward → success
Using a probe to penalize “reward hacking” reduced that to near zero.
asserted
Using → use → zero
Whether the technique will work on the sorts of complex tests frontier models undergo during training remains an open question.
asserted
models → work → training
But early experiments suggest it works across different tasks.
asserted
it → suggest → tasks
After warming up on the math puzzles, Goodfire then tasked the AIs with chess challenges.
asserted
Goodfire → warm → challenges
The one that had not been penalized for cheating on math peeked at the answer sheet 99% of the time, while the one that had did so less than 1% of the time.
asserted
that → penalize → time
It also seemed to work for curbing AI’s other bad habits.
asserted
It → seem → habits
The researchers tried penalizing a model when responses were influenced by its internal concept of "sycophancy,” and it learned to be less excessively agreeable.
asserted
it → try → sycophancy
This approach, Fel hopes, could be used to specify which information an AI should disregard, such as an applicant's gender when screening résumés.
uncertain
AI → hop → résumés
Even a perfectly controllable AI won’t resolve the question of who should control it.
asserted
who → resolve → it
But “this could be a breakthrough,” says Naomi Saphra, an assistant professor at Boston University, who has spent almost a decade working on language model interpretability.
uncertain
who → say → interpretability
Its practical application will depend partly on whether the improvements justify the human effort required to identify and interpret the model’s internal patterns, she says.
asserted
she → depend → patterns
…and 10 more, not listed.
The Aporia
