AI Interpretability Explained: How Researchers Actually Test a Model's "Thoughts"

Ask a large language model a question and it gives you an answer with total confidence. Ask it how it got there, and it will happily make something up — the model has no built-in ability to narrate its own internal process, because its internal process was never written in a language anyone can read. That gap is exactly what the field of AI interpretability exists to close, and the way researchers are closing it looks less like reading code and more like neuroscience.

Why You Can't Just "Read" a Neural Network

A modern language model's actual substance is a long list of numbers — weights — connecting artificial neurons across dozens of layers. Even a modestly sized open model has billions of these weights, and none of them were hand-written by an engineer. They emerged from training: the model read enormous amounts of text, repeatedly guessed the next word, and had its weights nudged a tiny amount every time it was wrong. Multiply that nudge by trillions of guesses and you get a system that works, but that nobody designed in the ordinary sense. There is no line of code that says "capital cities" or "sarcasm." Whatever those concepts look like inside the model, they're an emergent side effect of the training process, not an authored feature.

This is the core problem interpretability research is trying to solve: the model's behavior is observable, but the mechanism producing that behavior is not — unless someone builds the tools to go looking for it.

From Neurons to Features

Early attempts to understand these networks focused on individual neurons, the basic computational units. The hope was that a given neuron might reliably correspond to a single human-recognizable idea. That hope mostly didn't pan out. Individual neurons were often found to respond to several unrelated things at once, a problem researchers call polysemanticity — a neuron might fire for something related to, say, French text in one context and rhyming words in another, with no obvious shared thread connecting the two.

The more productive unit turned out to be something researchers call a feature: not a single neuron, but a pattern distributed across many neurons that lights up together. Anthropic's interpretability team made this concrete in a 2023 paper, "Towards Monosemanticity," which used a technique called dictionary learning to decompose a small model's internal activity into interpretable, concept-aligned features rather than raw neurons. A later 2024 follow-up, "Scaling Monosemanticity," extended the same method to Claude 3 Sonnet, a full production-scale model, and extracted millions of candidate features — a dramatic jump from toy demonstrations to something resembling a real map of a deployed system's internals.

One of the more memorable public demonstrations of this came from a 2024 research artifact Anthropic nicknamed "Golden Gate Claude." Researchers identified a feature that appeared to represent the concept of the Golden Gate Bridge and then artificially amplified it. The resulting model began bringing up the bridge in almost every response, regardless of what it was actually asked — tangible evidence that the feature wasn't just correlated with bridge-related text, but was doing causal work in producing it.

Watching the Model Take Two Steps, Not One

A separate and more recent strand of this research looks at whether models combine ideas internally rather than simply repeating memorized text. Anthropic's 2025 "On the Biology of a Large Language Model" work, and its companion piece "Tracing the Thoughts of a Large Language Model," walk through a deceptively simple example: ask a model to complete "the capital of the state containing Dallas is," and it answers "Austin." The lazy explanation is that the model simply memorized that exact sentence somewhere in its training data. The research shows something more interesting is actually happening — using a technique called attribution graphs, the researchers observed the model activate an internal representation corresponding to "Texas" after processing "Dallas," and only then activate features associated with naming a capital city, landing on "Austin." The model appears to be chaining two separate facts together internally — Dallas is in Texas, and Texas's capital is Austin — rather than regurgitating one long memorized string.

Testing the Theory Instead of Just Admiring It

The part of this research that actually earns the word "science" is what happened next: the researchers didn't stop at an interesting-looking diagram. They tested whether the intermediate "Texas" step was really doing causal work, by deliberately substituting it. Starting from a parallel prompt about Oakland instead of Dallas, they swapped in the internal representation for "California" in place of "Texas," while leaving the rest of the circuit alone. If the "state" step was genuinely driving the output, changing it should change the final answer — and it did. The model answered "Sacramento," California's capital, confirming that the intermediate concept wasn't decorative; it was load-bearing.

That swap test is the detail easiest to miss and most important to keep. A feature that merely correlates with a concept is a much weaker finding than a feature that, when deliberately altered, reliably changes the model's output in the predicted direction. The second kind of evidence is causal; the first kind is just a pattern that might coincidentally line up.

Why the Distinction Between Demonstration and Verification Matters

It's worth being explicit about what this research does and doesn't establish. These techniques work on specific case studies chosen because they illustrate a mechanism clearly — they are not yet a general-purpose readout of everything a large model is doing on every prompt. Interpretability researchers themselves are candid that attribution graphs capture only a partial, approximate view of a model's internal computation, and that scaling these methods to fully explain arbitrary model behavior remains unsolved. A compelling demonstration on one carefully chosen example is evidence, not proof of a completely solved problem.

That distinction matters for anyone consuming interpretability claims secondhand, whether in a research summary, a conference talk, or a short video. "Researchers found a feature that looks like X" is a weaker claim than "researchers found a feature that, when switched off or swapped, changed the output the way the X-hypothesis predicted." The field's real progress lives specifically in that second category, and it's worth checking which one you're being told about before treating a finding as settled.

Frequently Asked Questions

Is AI interpretability the same as explaining why a chatbot gave a specific wrong answer?

Not exactly. Interpretability research mostly studies internal mechanisms using controlled, often simplified examples. Explaining one specific real-world wrong answer in full detail is a related but much harder and less solved problem.

Does finding a "Texas feature" mean the model understands geography the way a person does?

No — it means the model has internal representations that function analogously to geographic facts for the purpose of predicting text, and that those representations causally affect outputs in controlled tests. Whether that constitutes "understanding" is a separate philosophical question these papers don't claim to settle.

Who is doing this kind of research?

Anthropic's interpretability team publishes much of the most detailed public work in this specific area, but mechanistic interpretability is an active academic field with researchers at multiple universities and labs contributing related methods and findings.

Why does any of this matter if the model still works fine without it?

Because a system that behaves well in testing can still fail in ways nobody anticipated. Understanding the actual mechanism — rather than just the observed behavior — is what lets researchers predict and guard against failure modes before they show up in the real world, instead of only after.