# Discovering Transformer Circuits via a Hybrid Attribution and Pruning Framework

arXiv: https://arxiv.org/abs/2510.03282 · HTML: https://mlsanity.com/paper/2510.03282/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [Low-Complexity Probing via Finding Subnetworks](https://mlsanity.com/paper/2104.03514/) (2021)
- [Finding Transformer Circuits with Edge Pruning](https://mlsanity.com/paper/2406.16778/) (2024)
- [SlimGPT: Layer-wise Structured Pruning for Large Language Models](https://mlsanity.com/paper/2412.18110/) (2024)
- [Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition](https://mlsanity.com/paper/2407.00886/) (2024)
- [Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop](https://mlsanity.com/paper/1904.04063/) (2019)
- [EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification](https://mlsanity.com/paper/2502.06852/) (2025)
- [An overview of 11 proposals for building safe advanced AI](https://mlsanity.com/paper/2012.07532/) (2020)
- [Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models](https://mlsanity.com/paper/2410.01434/) (2024)
- [Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gates](https://mlsanity.com/paper/2505.10039/) (2025)
- [Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining](https://mlsanity.com/paper/2407.19126/) (2024)

## Built on by (most cited first, 4 of 4)

- [Mechanistic Interpretability as Statistical Estimation: A Variance Analysis](https://mlsanity.com/paper/2510.00845/) (2025)
- [Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decomposition](https://mlsanity.com/paper/2601.12879/) (2026)
- [Emergent Alignment](https://mlsanity.com/paper/2606.19527/) (2026)
- [RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons](https://mlsanity.com/paper/2608.24758/) (2026)

## Abstract (arXiv)

Interpreting language models often involves circuit analysis, which aims to identify sparse subnetworks, or circuits, that accomplish specific tasks. Existing circuit discovery algorithms face a fundamental trade-off: attribution patching is fast but unfaithful to the full model, while edge pruning is faithful but computationally expensive. This research proposes a hybrid attribution and pruning (HAP) framework that uses attribution patching to identify a high-potential subgraph, then applies edge pruning to extract a faithful circuit from it. We show that HAP is 46% faster than baseline algorithms without sacrificing circuit faithfulness. Furthermore, we present a case study on the Indirect Object Identification task, showing that our method preserves cooperative circuit components (e.g. S-inhibition heads) that attribution patching methods prune at high sparsity. Our results show that HAP could be an effective approach for improving the scalability of mechanistic interpretability research to larger models. Our code is available at https://anonymous.4open.science/r/HAP-circuit-discovery.