# Data-driven Circuit Discovery for Interpretability of Language Models

arXiv: https://arxiv.org/abs/2605.09129 · HTML: https://mlsanity.com/paper/2605.09129/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 9 May 2026.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis](https://mlsanity.com/paper/2305.15054/) (2023)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs](https://mlsanity.com/paper/2402.07321/) (2024)
- [Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs](https://mlsanity.com/paper/2506.09047/) (2025)
- [Discovering Variable Binding Circuitry with Desiderata](https://mlsanity.com/paper/2307.03637/) (2023)
- [Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context](https://mlsanity.com/paper/2510.06182/) (2025)
- [All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens](https://mlsanity.com/paper/2509.09650/) (2025)
- [Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones](https://mlsanity.com/paper/2507.00322/) (2025)

## Built on by (most cited first, 2 of 2)

- [ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation](https://mlsanity.com/paper/2608.22559/) (2026)
- [Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuit](https://mlsanity.com/paper/2608.27254/) (2026)

## Abstract (arXiv)

Circuit study aims to explain how language models (LMs) implement a task by localizing and interpreting a circuit: a computational subgraph responsible for a specific model's behavior. Existing methods for circuit localization are largely hypothesis-driven: they define a task through a dataset, assume that all examples in the task are implemented by the same mechanism, and apply circuit discovery to obtain a single circuit. This workflow does not account for mechanistic diversity within a task, where different examples may rely on different mechanisms. We first show that when a dataset mixes tasks processed through distinct mechanisms, current circuit discovery can combine these mechanisms into a single circuit while retaining faithfulness comparable to circuits discovered on each task separately. Across four tasks and three models, we further provide evidence for such mechanistic diversity within individual tasks. To account for this diversity, we propose Data-driven Circuit Discovery (DCD), which modifies the circuit-discovery workflow by adding a grouping step before the discovery. The grouping step aims to partition the dataset into subsets of examples that rely on similar mechanisms. We instantiate this step with clustering over per-example attribution patterns (DCD-Cluster). Across four tasks and three models, DCD-Cluster decomposes the dataset into multiple subsets whose circuits exhibit distinct faithfulness patterns, providing useful hypotheses about mechanistic differences among examples.