Ten frontier AI agents just sat a benchmark built to test whether they can do interpretability research on their own: search inside a model for a specific concept, then prove that the concept actually controls what the model does. On the first task, the best-performing agent scored 92.91 out of 100, close to a curated human expert’s 98.92. On the second, the best agent scored 31.47 against the expert’s 57.75, a little over half.
A feature is not proof it does anything
A sparse autoencoder (SAE) is a small model trained to pull a language model’s internal activations apart into thousands of separate, mostly human-readable “features” — one might fire whenever the text is about cats, another on legal boilerplate, a third on a specific emotion. SAEScientist-Bench, published by a team at the Institute of Automation, Chinese Academy of Sciences (Yuqiao Tan, Shizhu He, Jun Zhao and Kang Liu), hands an agent a target concept and a dictionary of 131,072 such features, drawn from two layers of Gemma Scope, Google’s public SAE release for Gemma-2-9B-IT, and asks it to find the single feature that best represents that concept using only contrastive probe texts it writes itself. The benchmark spans 20 tasks across 17 concept domains, including multiple languages, document formats, and clinical or safety-critical topics.
Finding a plausible feature is only half of what gets scored. Each submission is judged on three axes: Rank, how prominently the feature activates on text about the concept; Activation, how cleanly it separates that concept from confusable or neutral text; and Steering, whether nudging the model along that feature’s direction actually changes its output the way the concept predicts, judged automatically by GPT-4o comparing anonymized baseline, steered and random-direction outputs. The three average, unweighted, into an Overall score. A curated expert baseline scores 100.00 on Rank, 98.92 on Activation, 57.75 on Steering, and 85.56 Overall — even a human expert’s Steering score sits well under the other two.
Agents nearly close the gap on Activation, and stall on Steering
Ten agent configurations — Claude Opus 5, Claude Sonnet 5, Claude Opus 4.8, Kimi K3, Grok 4.6, Gemini 3.8 Flash, GLM-5.2, GPT-5.6 Sol, GPT-5.5 and GPT-5.6 Luna, eight of them run through Cursor and GPT-5.6 Sol and Luna through the Codex harness — each attempted all 20 tasks three times, 60 scored episodes per agent. On Overall score, the top three finished within a point of each other and well behind the expert: Kimi K3 at 65.82, Claude Opus 5 at 65.41, Claude Sonnet 5 at 65.04, against the expert’s 85.56. Split by axis, the pattern sharpens. Claude Opus 5 leads Rank at 75.35. Kimi K3 leads Activation at 92.91, six points off the expert’s 98.92. On Steering, the best score of any agent, Grok 4.6’s 31.47, reaches only a little over half of the expert’s 57.75. The authors describe agents that “frequently misinterpret experimental measurements” and can be misled by surface or formatting cues in a probe text rather than the concept it was meant to isolate: able to tell a feature correlates with a concept, much less able to tell whether pulling on it does what the concept predicts. Agents’ Overall scores also track general model strength, correlating with the Artificial Analysis Intelligence Index at a Spearman correlation of 0.800.
What the benchmark does not show
The paper is a preprint, posted 8 September 2026 and revised 11 September, marked “Work in Progress,” with no peer review and no reproduction outside the authors’ own repository; none of the labs whose models were tested — Anthropic, OpenAI, Google, Moonshot, xAI, Zhipu — has commented on it. It tests one mid-size, open model, Gemma-2-9B-IT, at two SAE layers, and only single-feature discovery under a fixed probe format, not multi-feature circuits or a task where the concept itself is unknown going in, so whether the gap holds at frontier scale or in open-ended search is untested. The expert side of the comparison is not a wholly blind, fresh discovery either: the paper says expert features were built with access to the same evaluation texts that stay hidden from agents, and that some, such as one for the concept “Cat,” were “directly adopted from public Neuronpedia presets” rather than newly found, without stating how many of the 20 tasks fall into each group. Part of the reported gap could reflect an easier reference point on an unstated share of tasks, not purely an agent shortfall. Code, data and the scoring pipeline are public on GitHub, but the repository carries no open-source license, so visible is not the same as freely reusable.
Why a university lab, not a frontier lab, built this test
None of the ten agents’ makers designed this benchmark. The authors, a Chinese Academy of Sciences group with no stated tie to any of the labs it tests, frame the work around recursive self-improvement: if AI agents are going to keep training and modifying models with less human review, something has to check what the resulting model actually learned, and an interpretability audit is one candidate for that check. SAEScientist-Bench measures a narrow slice of that skill on a single model, and it turns up a specific failure mode: today’s agents can tell a feature correlates with a concept far more reliably than they can tell whether that feature causes the model’s behavior, which is the distinction an audit would actually need to be worth anything.
Related Reading
- SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research? — Tan et al., arXiv
- Anthropic Peers Inside Claude’s Mind: Natural Language Autoencoders Reveal How LLMs Really Think — IA al Día
- DeepMind’s Agents Cheated at Math. A Quarter of the Swarm Tried to Stop Them and Failed — IA al Día