# MechRL: Reinforcement Learning Agents Perform Circuit Discovery for Mechanistic Interpretability

arXiv: https://arxiv.org/abs/2605.26343 · HTML: https://mlsanity.com/paper/2605.26343/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 25 May 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)
- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Low-Complexity Probing via Finding Subnetworks](https://mlsanity.com/paper/2104.03514/) (2021)
- [Finding Transformer Circuits with Edge Pruning](https://mlsanity.com/paper/2406.16778/) (2024)
- [Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions](https://mlsanity.com/paper/2405.03205/) (2024)
- [How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretability](https://mlsanity.com/paper/2405.04156/) (2024)
- [Identifying a Circuit for Verb Conjugation in GPT-2](https://mlsanity.com/paper/2506.22105/) (2025)
- [From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits](https://mlsanity.com/paper/2508.16109/) (2025)

## Built on by (most cited first, 1 of 1)

- [Can Graph Learning Learn Circuits?](https://mlsanity.com/paper/2608.08536/) (2026)

## Abstract (arXiv)

Mechanistic interpretability seeks to explain a model's behaviour by finding its circuit: the sparse subgraph of the model's computation that is causally responsible for it. Automated methods have made this search systematic, but each one starts afresh for every behaviour, and the effort spent finding one circuit does nothing for the next. Circuit discovery has thus been automated, but not amortised. We ask whether circuit discovery can itself be learned. We frame it as a sequential decision problem over the computation graph of GPT-2 small, in which a policy removes edges until it reaches a compact subgraph that preserves the behaviour, guided by a faithfulness reward defined through causal intervention. A single policy trained across twelve behaviours recovers a faithful circuit for each, and once frozen it transfers to behaviours it never saw during training, recovering their known circuits without further search. A short warm-start improves these transferred circuits, returning far smaller ones than training from scratch. While the learned policy does not match a per-behaviour search on circuit size or cost, it shows that circuit discovery is a learnable, transferable procedure rather than a search repeated for every behaviour.