# Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decomposition

arXiv: https://arxiv.org/abs/2601.12879 · HTML: https://mlsanity.com/paper/2601.12879/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 23 of 23)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://mlsanity.com/paper/2307.09458/) (2023)
- [Finding Transformer Circuits with Edge Pruning](https://mlsanity.com/paper/2406.16778/) (2024)
- [Weight-sparse transformers have interpretable circuits](https://mlsanity.com/paper/2511.13653/) (2025)
- [Defining and Quantifying the Emergence of Sparse Concepts in DNNs](https://mlsanity.com/paper/2111.06206/) (2021)
- [Explaining Generalization Power of a DNN Using Interactive Concepts](https://mlsanity.com/paper/2302.13091/) (2023)
- [Automatically Identifying Local and Global Circuits with Linear Computation Graphs](https://mlsanity.com/paper/2405.13868/) (2024)
- [Opening the AI black box: program synthesis via mechanistic interpretability](https://mlsanity.com/paper/2402.05110/) (2024)
- [Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations](https://mlsanity.com/paper/2502.18147/) (2025)
- [Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Models](https://mlsanity.com/paper/2305.01939/) (2023)
- [The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks](https://mlsanity.com/paper/2405.10928/) (2024)
- [Discovering Transformer Circuits via a Hybrid Attribution and Pruning Framework](https://mlsanity.com/paper/2510.03282/) (2025)

## Abstract (arXiv)

Extracting sparse circuits from billion-parameter transformers is constrained by O(2n) search cost and pervasive feature reuse across co-active pathways. Hierarchical Attribution Graph Decomposition (HAGD) addresses this through four stages: cross-layer transcoder training, spectral coarsening of attribution graphs, graph-neural-network (GNN)-guided hierarchical traversal, and causal intervention verification, reducing worst-case complexity to O(n2 log n). Per-layer transcoders trained on the RedPajama corpus yield monosemantic dictionaries; gradient-activation products form weighted attribution graphs; normalized-Laplacian spectral clustering builds multi-resolution hierarchies; an attention-based GNN assigns circuit-membership scores at successive coarsening stages. Evaluation spans GPT-2 (117M-774M), Pythia (1.4B-6.9B), and Llama (7B-70B) across modular arithmetic, parity computation, integer sorting, coreference resolution (WinoGrande), commonsense reasoning (HellaSwag), and factual recall. Behavioral preservation reaches 91% (±2.3%) on modular arithmetic with 49-347-node circuits, while ACDC exhausts memory beyond 1.4B parameters. Cross-architecture transfer coefficients span 0.38-0.82, with within-family pairs (Llama-7B → Llama-70B) attaining 0.82. Limitations include omitted attention-head circuits, 15-20% unexplained reconstruction variance, ablation-based validation circularity, and uncertain interpretability of circuits exceeding several hundred nodes.