The J-Space: How I Learned To Read An LLM's Mind

By Maia Salti

Generated by Nano Banana

Last week, Anthropic published a paper on a discovery they made regarding Claude's internal reasoning they call the J-space. It seems to be the steps that Claude works through before it commits to a final word: the closest thing we've come to seeing the inside of an LLM's "brain."

Although it holds a median of only 6–7% of a concept's representation inside the model and never more than about a tenth of the model's activity at any layer, if you switch it off, Claude's multi-step reasoning collapses to almost nothing. Fluent speech and simple recall remain intact.

I quite liked the video that Anthropic released with the research post. It's a long research post though, so I thought I'd write a summary of the parts I considered the coolest and how I interpreted the mathematics of the J-space.

The Transformer

To understand how they found the J-space, one must understand the basic architecture of a transformer. I wrote a full article about how LLMs work if you want to check it out here, but here's a brief explanation:

Every large language model, including Claude, runs the same process: input embedding → layer 1 → layer 2 → … → final layer → unembedding matrix (W_U) → softmax.

As I love my analogies, I will ask you to picture it as a factory building with 50 floors. On each floor, the workers speak a different technical language from the other floors. An iPhone is in production, and it starts from the bottom floor, and each floor modifies the iPhone as it ascends the floors.

When the iPhone is on any one of the floors besides the last, say floor 37, it doesn't look like a phone yet, and it's only interpretable to the people on that floor. At the very top floor, there is an official dictionary that tells you exactly how to use the iPhone, but that dictionary obviously only works once the iPhone is fully built, not at any other floor. So what if you wanted to see what's going on inside of the phone at floor 37? Well, you couldn't, until now.

Let me translate this to the LLM structure: the floors are the model's layers and the iPhone is the activation vector that gets modified through the floors. The unembedding matrix (W_U) is the dictionary that lets you translate the activation vector (a vector of raw numbers) to logits (a raw score for each word). Those logits get passed through softmax, which squashes that list of scores into probabilities that add up to 1. Whichever word ends up with the highest probability is usually the one the model says next. If you apply the unembedding matrix (W_U) to the activation vector at any of the middle layers, it won't translate properly, as those layers speak a different language.

Anthropic discovered that if you tweaked how the iPhone was made at every floor one by one and saw how that affected the final iPhone, you could read what each floor is contributing to the final phone.

Okay maybe I ran on too long with my analogy. The point is this: the Jacobian lets you read a mid-layer vector in the output's language by measuring which words it's influencing downstream. That reveals the concepts the model is holding at that layer, before it has fully formed its output. This, at least in my opinion, is the closest thing we've come to seeing how the model "thinks."

What a Jacobian Is

The tool Anthropic used to build the translation comes from ordinary calculus: the Jacobian.

In maths, a Jacobian is a matrix of how much each output moves with respect to each input. Move input x by a little bit, and it tells you how y moves, and by how much.

It's the flat line you'd draw to approximate a curve near one point, the gradient, just generalized to many numbers at once instead of one: the multivariable tangent line. The best linear approximation of how a function stretches, rotates, or squishes space near a given point.

Illustrative, not a real model output. It shows the idea: pick a middle layer, fire a straight-line guess from there, and see how close it lands to where the model actually ends up. Slide the layer control and notice the estimate gets better the closer you get to the end.

How the Jacobian Inserts Into the Stack

Anthropic built one of these translators for every layer of Claude, the transport matrix:

$J_\ell = \mathbb{E}_{t,\ t' \ge t,\ \text{prompt}}\left[\frac{\partial h_{\text{final}, t'}}{\partial h_{\ell, t}}\right]$

To explain it simply: how much does adjusting the activation at layer ℓ, position t, move the final-layer activation at position t', averaged (E, for expectation) across a bunch of different prompts and future positions. J_l is averaged over roughly a thousand pretraining-like prompts, not computed for whatever you happen to be asking.

And it's not only about the very next word. It accounts for where the model is going across the words ahead: the general direction it's heading and not just its immediate next guess.

Multiply a middle-layer vector by its J_l, and you carry it into the final layer's basis (language).

Jacobian to J-Lens

Once that vector has been carried into final-layer coordinates, you run it through the normal last step: normalize, take the dot product with W_U, and softmax:

$\text{lens}(h_\ell) = \text{softmax}\big(W_U\ \text{norm}(J_\ell h_\ell)\big)$

The output is a ranked list of words the vector is projected to produce downstream.

Anthropic calls this the J-lens, and the small slice of representation it's reading is the J-space: the model's private, mid-process "thinking," sometimes revealing what it's silently considering even when it never says it out loud.

You can run the J-lens yourself on real prompts. Neuronpedia built a live demo which I've been playing with.

Why the Middle Layers

If you apply the J-lens at every middle layer sequentially, you can sometimes see the progression of the model's multi-step reasoning.

Anthropic groups the layers into three rough bands. Early layers are sensory, still close to the raw input tokens and not saying much yet. Late layers are motor, already collapsed toward the word the model is about to output.

The middle is the workspace, running from roughly 30% of the way through the network to shortly before the end. This is where words that seem relevant to the eventual output start showing up, well before they're actually said.

For example, a multi-step arithmetic problem: (4 + 17) × 2 + 7. The final answer is 49, but the model has to pass through two intermediate results, 21 and then 42, to get there.

Illustrative, not a real model output — but it mirrors what the paper describes: watch the readout at each layer as you scrub (or hit play), and you'll see 21 surface first, then 42, then 49, each one appearing a few layers before the last. The reasoning steps arrive in order, not all at once at the end.

Proving It's Not Just a Correlation

Anthropic ran causal swap experiments and took the J-space representation of one concept mid-computation, replaced it with the representation of a different concept, and let the rest of the network run normally.

If the J-space were only correlated with the model's reasoning and not the answer, swapping it shouldn't change anything Claude actually outputs. Instead, it behaved like the thing driving the answer, and it did that in five ways:

Verbal report — swap the concept Claude is "thinking about" mid-sentence (banana → elephant), and when you ask it what it was just thinking about, it tells you the swapped concept, not the original one.

Directed modulation — swap a concept while Claude is in the middle of writing something unrelated, like a sentence about a painting, and the swap steers what it writes next, even though nothing about the visible prompt changed.

Internal reasoning — swap a fact Claude is silently relying on (Mars → Earth) and a completely different, downstream question ("what color is that planet?") gets answered based on the swapped fact instead of the real one.

Flexible generalization — swap "France" for "China," and Claude's answers about the capital, the language, the continent, and the currency all shift together, consistently. As if you'd changed the actual subject of the sentence, not just one detail.

Selectivity — ablate the J-space entirely, across a whole battery of tasks, and simple stuff (parsing input, recalling facts, speaking fluently) keeps working fine, while anything that requires internal reasoning or multi-step inference drops out. This is the same ablation result from the intro: it's what tells you the J-space isn't just along for the ride: it's doing the actual thinking.

Why This Matters

This doesn't mean that Anthropic has "solved" mechanistic interpretability. But it does mean that researchers now have a lens that lets them watch a model's intermediate reasoning steps appear layer by layer, before the final answer is spoken.

Considering an LLM's internal reasoning has been a blackbox since it was created, reading a rough draft of the thought is a significant step forward and makes me incredibly hopeful for the discoveries to come.

I do recommend reading Anthropic's research post, and if you want a more technical read, the full paper has the methodology and additional experiments I didn't talk about here.

If you're experienced with interpretability research and think I oversimplified something, please email me at maia.salti@gmail.com. I'd love to learn as much as I can.