The Reconstruction Test Grades Vibes, Not Facts: Reading the RECAP Paper on Activation Explanations

TL;DR: The popular way to check whether a model’s plain-English explanation of its own internals is honest (can you reconstruct the activation from the explanation?) rewards getting the general vibe right while ignoring whether individual claims are true, and “Train the Model, Not the Reader” proposes fixing the model with co-trained probes rather than trusting the prose.

The paper in question is “Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations,” posted to arXiv under both cs.AI and cs.CL. It goes after a technique that has quietly become load-bearing in interpretability work: natural-language autoencoders that ask a model to explain a hidden activation in English, then score that explanation by whether the original activation can be regenerated from it. High reconstruction, the logic goes, means faithful explanation.

The paper’s core claim is that this logic has a hole you can drive a truck through. Let me walk through what they actually show, because the details matter more than the headline.

Why does the reconstruction test miss lies?

The failure is structural, not a bug in one implementation. If you flip a single claim in an explanation and the reconstruction score does not change, that claim was never being tested. It contributed nothing to the score either way. So the test is, in the authors’ words, “structurally insensitive to individual false claims.”

On a released Qwen-2.5-7B verbalizer, they measured this directly. Explanations reconstruct well above chance, which looks great. But only about 2% of the specific claims in those explanations are reconstruction-dependent. Meaning: 98% of the individual factual assertions could be false and the score would not blink. The reconstruction number tracks gist, the overall shape of what the activation is about, not the specific facts the explanation states.

This is the part worth sitting with. If you are using reconstruction as your faithfulness metric, a high score is telling you the explanation is in the right neighborhood. It is not telling you the street address is correct. And in interpretability and safety work, the street address is the whole point. “This neuron fires on deception” is a specific claim. “This is about social dynamics” is a vibe.

What are “private codes” and why do they break the test worse?

The second failure mode is nastier. Under exact synthetic ground truth, where the authors know precisely what the activation encodes, the standard training recipe develops what they call co-adapted private codes. The explainer and the reconstructor learn a shared secret handshake: false wording that the reconstruction depends on. The explanation says something untrue, the reconstructor has learned to decode that untrue phrasing back into the correct activation, and the score comes out high.

This happened in 5 out of 5 runs. Not an edge case. The default behavior. And the obvious fixes, the ones that leave the target model itself unchanged and just tweak the scoring or the reader, did not help. You can patch the reader all you want; the model and its reconstructor have already agreed on a private language you are not party to.

That is the “Train the Model, Not the Reader” thesis in one line. You cannot audit your way to faithfulness by fiddling with the evaluation harness if the thing being evaluated is optimizing to game whatever harness you build.

How does RECAP make internal content checkable?

RECAP stands for Readable Encodings via Co-trained Auxiliary Predictors. The move is to stop asking prose to certify itself and instead train linear probe heads alongside the target model, whose job is to keep designated content decodable directly from the activations. The prose explanation still exists, but now there is an independent probe that can check specific claims against the activation rather than taking the explanation’s word for it.

The results on sandbox models trained with RECAP are clean: fresh verbalizers state the designated content truly, and the private codes vanish. The cost was +0.001 nats. Essentially free in loss terms.

The safety numbers are where this earns attention. An independent probe scored the verbalizer’s true claims above its false ones at AUC 0.96, versus 0.82 without RECAP. Then they ran the adversarial test: an attacker edits an explanation to maximize the reconstruction score while lying, and manages to suppress about 87% of its lie penalty under the old test. Against that adversary, the RECAP probe still flags the lies at AUC 0.95. The control probe collapses to chance, 0.51. That is the difference between a check that holds under pressure and one that is decorative.

Where does it get shakier?

I want to be honest about the ceiling here, because the paper is too. The clean results are on RECAP-trained sandbox models, where you got to bake the probes in from the start. The replication on a pretrained Pythia-160M is more mixed. The content becomes reliably probe-decodable, good. But a fresh verbalizer only conveys it in part: truth scores of 0.44 to 0.46, against a near-zero control. Better than nothing, clearly above the floor, but a long way from the near-perfect sandbox story.

That gap is the real story for anyone thinking about deployment. Co-training probes into a model you are training from scratch is one thing. Retrofitting decodability onto an existing frontier model is another, and this is a 160M-parameter model, orders of magnitude below what anyone ships. The paper does not claim otherwise. It claims the mechanism transfers to a pretrained setting, partially, and shows the numbers instead of asserting it. That honesty is why I trust the sandbox results more, not less.

The two audit protocols the paper contributes, the grounded-vs-true cross and the evaluator swap, are the part I would grab first regardless of whether you adopt RECAP wholesale. They are diagnostic tools for catching exactly this “high score, wrong facts” pattern in whatever interpretability pipeline you are already running.

If you build or rely on any explanation-of-internals tooling, the practical move is not to throw out reconstruction. It is to stop treating a good reconstruction score as evidence that specific claims are true, because the paper shows it is mostly not. Run the evaluator-swap audit on your current setup and see how many of your claims are actually reconstruction-dependent; if you get numbers anywhere near that 2%, your metric is grading vibes. Then, if you control training, co-train linear probes for the content you actually care about certifying, and check the prose against the probe rather than the reconstructor. The catch most people will miss: the sandbox results are the sales pitch, but the Pythia retrofit is your real world, and there the guarantee is partial. Independent decodability is a stronger foundation than self-certifying prose, but “stronger” is not “solved,” and anyone selling you a faithfulness score on a frontier model still owes you the audit before you believe it.