mlsanityData to 7 Oct 2026

Automated Interpretability and Feature Discovery in Language Models with Agents

arXiv 2605.01555 ↗First posted 2 May 2026cs.CL cs.AI cs.HCcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce an autonomous multiagent framework for mechanistic interpretability that automates both explaining and finding internal features in large language models. The system runs two coupled loops: (1) explanation refinement, where an agent proposes competing hypotheses and iteratively tests them with targeted prompt controls and a multi-metric evaluation; and (2) feature discovery, where an agent generates prompt sets, constructs a k-nearest-neighbor graph in activation space, and retrieves candidate features using statistical separability and semantic coherence criteria. On Gemma-2 family models and MLP neurons in weight-sparse transformers, our agent improves over one-shot auto-interpretations, discovers language-specific and safety-relevant features, and produces auditable explanation traces, showing that agent-driven empirical loops yield sharper and more falsifiable explanations than one-shot labels.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 40 references that are in the map. First posted on arXiv 2 May 2026.

Builds on

21 of 40 references.

References · 21

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  3. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  9. 2024Scaling and evaluating sparse autoencoderscited by 530
  10. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  11. 2018From Louvain to Leiden: guaranteeing well-connected communitiescited by 379
  12. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
9 more
  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  2. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  3. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  4. 2024A Multimodal Automated Interpretability Agentcited by 62
  5. 2025Open Problems in Machine Unlearning for AI Safetycited by 49
  6. 2025Weight-sparse transformers have interpretable circuitscited by 40
  7. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 18
  8. 2025Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systemscited by 15
  9. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?cited by 1
  2. 2026Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligencecited by 1
  3. 2026SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models
  2. 2026Pitfalls in Evaluating Interpretability Agents
  3. 2026MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations
  4. 2023Explaining Agent Behavior with Large Language Models
  5. 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation
  6. 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models
  7. 2026Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs
  8. 2026Mechanistic Interpretability for Large Language Model Alignment: Progress, Challenges, and Future Directions