Can AI Agents Do Their Own Interpretability Research? A New Benchmark Says Not Yet

TL;DR: A new benchmark called SAEScientist-Bench shows frontier agents can genuinely hunt down interpretable features in a model, but they still lag human experts badly on causal steering and, worse, they misread their own experiments.

Most of the excitement around recursive self-improvement points one direction: agents that automate model training, run experiments, tune hyperparameters, push the loop faster. The paper worth reading this week points at the part everyone skips. If agents are going to build and improve models with less human oversight, someone has to audit what those models actually learned. That someone might also be an agent. Can it do the job?

The primary source here is SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?, posted across arXiv’s cs.AI, cs.CL, and cs.LG listings, with code at github.com/Trae1ounG/SAEScientist. It sets up a concrete test: give an agent a target concept, hand it real interpretability tooling, and see if it can find the feature a human expert would find.

What is the benchmark actually testing?

The setup is specific, which is why it matters. The agents work inside Gemma-2-9B-IT using Gemma Scope, a dictionary of more than 131,000 sparse autoencoder features. A sparse autoencoder, in plain terms, is a tool that pulls apart the tangled internal activations of a model into a large set of individual features that are easier to interpret. One feature might fire on “legal language,” another on “Python code,” another on some concept you did not expect.

The task: given a target concept, the agent has to design contrastive probes (texts that should activate the concept versus texts that should not), search that 131K feature dictionary, and identify the single best feature for the concept. The paper evaluates the agent’s pick against curated expert reference features anchored on Neuronpedia, along three axes: activation rank, concept selectivity on contrastive texts, and causal steering. That last one is the real test. Steering means actually pushing the feature to change model output, not just spotting that it lights up.

This is a good benchmark design because it separates two things people usually blur. Recognizing a feature is not the same as controlling it. You can point at the “sarcasm” neuron all day; making the model reliably more or less sarcastic on command is a harder claim, and the benchmark grades them separately.

How well did the agents do?

Across 10 agent configurations and 20 tasks, the paper reports that frontier agents “demonstrate genuine discovery capabilities.” That is not throat-clearing. Different agents led different evaluation dimensions, which suggests the capability is real and not a fluke of one lucky model. On separating a target concept from contrastive controls, the paper says agents approached expert levels. So on the recognition side, the machines are getting close.

Then the gap. On causal generation steering, agents lagged the expert baseline substantially. This is the part that would matter most for actual safety work. If you want to use interpretability to catch and correct undesired behavior, you need the causal handle, not just the descriptive label. The benchmark is telling us agents can describe a model’s internals reasonably well but cannot yet reliably intervene on them.

The most useful finding, and the one I would flag to anyone building agentic research pipelines, is buried in the analysis. The agents “frequently misinterpret experimental measurements.” They could design good contrasts to rule out spurious feature candidates, which is genuine scientific reasoning. But then they read their own results wrong. That is a specific, familiar failure. It is the intern who runs the right experiment and draws the wrong conclusion from the numbers.

Why does this matter beyond interpretability?

The paper frames the missing pillar clearly: RSI research has automated the training pipeline but not the post-hoc monitoring and auditing that would tell you whether the resulting model is safe and aligned. You cannot close the loop on autonomous AI R&D if the audit step still requires a human every time. So the question “can an agent do interpretability” is really “can an agent check the work of another agent.” The honest answer from this benchmark is: partially, and not on the part that matters most.

There is a broader lesson for anyone deploying agents on any evaluation-heavy task, not just interpretability. The failure mode here is not that the agent could not do the work. It ran the probes. It searched the space. The failure was in interpreting evidence. That maps directly onto agent workflows in analytics, QA, code review, anything where the agent produces a measurement and then has to decide what it means. The measurement is often fine. The judgment on top of it is where things break.

I would push back gently on one thing, and the paper is careful about this itself, so it is more a caution for readers. This is one benchmark, on one model (Gemma-2-9B-IT), with one SAE dictionary. “Frontier agents demonstrate genuine discovery capabilities” is a claim about this setup. It does not tell you agents can do interpretability on frontier-scale models with more entangled features, which is exactly where you would want the audit most. The gap the paper measures could be a floor, not a ceiling, once the models under audit get bigger.

What should a builder take from this?

If you are wiring agents into a research or QA loop, the takeaway is to stop trusting the interpretation step and start grading it separately from the execution step. This benchmark did exactly that and found the split. Your own pipeline probably has the same split hiding in it.

Concretely: when an agent runs an experiment and reports a conclusion, log the raw measurement alongside the conclusion, and spot-check whether the conclusion follows from the number. The SAEScientist-Bench result predicts that is where you will find silent errors, not in whether the agent ran the right test. Build a verification layer that re-reads results rather than assuming the agent’s summary is correct.

The catch most readers will miss: this is genuinely encouraging for descriptive interpretability and genuinely sobering for the safety use case. Recognizing a feature is not controlling it, and the whole promise of interpretability-for-alignment lives in the control. Agents are closest to expert on the part that is easiest and furthest on the part that would actually let you steer a misbehaving model. If you were hoping to hand the audit step to an agent and walk away, this paper says: not yet, and specifically not on causal steering. Keep the human on the intervention. Let the agent do the search.