Can Graph Learning Learn Circuits?
Not on the map. Why?
Abstract (arXiv)
Circuit localization is a mechanistic interpretability task whose goal is to identify a sparse subgraph of a transformer's computation graph sufficient to reproduce a particular behavior. Most established methods localize circuits independently for each model--task pair. We instead frame circuit localization as a graph machine learning problem in which the edges of a computation graph represent computational pathways, and graph neural networks (GNNs) model interactions among these pathways. We introduce Graph Circuit Learning (GCL), a supervised, amortized framework that trains a GNN across multiple model--task pairs and applies it to unseen cases. To provide sufficient data, we augment the InterpBench benchmark with additional cases derived from the TracrBench programs. Of the 14 evaluated GCL configurations, the highest scored a median edge AUROC of 0.902 (interquartile interval [0.861, 0.942]) on the 16 original held-out InterpBench cases. This is close to the published InterpBench median of 0.910 for EAP-IG while remaining below ACDC's 0.959. Removing all message-passing edges reduces the median to 0.825. We also adapt PGExplainer, a GNN explainability method, to circuit localization, obtaining a median edge AUROC of 0.858 on the same cases. These preliminary results suggest that graph machine learning offers a natural and potentially powerful perspective on circuit localization, and we hope this perspective encourages closer exchange between the two communities.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 9 Aug 2026.
Builds on
51 of 59 references.
References · 51
- 2017Attention Is All You Needcited by 59,932
- 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2020Rethinking Attention with Performerscited by 1,584
- 2017Deep Setscited by 1,394
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2019GNNExplainer: Generating Explanations for Graph Neural Networkscited by 735
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2020Parameterized Explainer for Graph Neural Networkcited by 423
- 2021On Explainability of Graph Neural Networks via Subgraph Explorationscited by 280
- 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
- 2021Thinking Like Transformerscited by 182
38 more
- 2020Interpreting Graph Neural Networks for NLP With Differentiable Edge Maskingcited by 152
- 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
- 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
- 2021Generative Causal Explanations for Graph Neural Networkscited by 124
- 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
- 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
- 2023Edge Directionality Improves Learning on Heterophilic Graphscited by 88
- 2024GFT: Graph Foundation Model with Transferable Tree Vocabularycited by 82
- 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
- 2023Permutation Equivariant Neural Functionalscited by 69
- 2021Low-Complexity Probing via Finding Subnetworkscited by 63
- 2024Finding Transformer Circuits with Edge Pruningcited by 58
- 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
- 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
- 2025PyG 2.0: Scalable Learning on Real World Graphscited by 49
- 2023Learning Transformer Programscited by 40
- 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 39
- 2024Explorations of Self-Repair in Language Modelscited by 35
- 2022MSGNN: A Spectral Graph Neural Network Based on a Novel Magnetic Signed Laplaciancited by 28
- 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
- 2022Transformers over Directed Acyclic Graphscited by 23
- 2022PyTorch Geometric Signed Directed: A Software Package on Graph Neural Networks for Signed and Directed Graphscited by 18
- 2024Compact Proofs of Model Performance via Mechanistic Interpretabilitycited by 17
- 2025Equivariance Everywhere All At Once: A Recipe for Graph Foundation Modelscited by 17
- 2026A Survey of Weight Space Learning: Understanding, Representation, and Generationcited by 15
- 2025Generalized Linear Mode Connectivity for Transformerscited by 14
- 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gatescited by 8
- 2025Beyond Components: Singular Vector-Based Interpretability of Transformer Circuitscited by 6
- 2026Billion-Scale Graph Foundation Modelscited by 5
- 2023Improving Graph Neural Networks on Multi-node Tasks with the Labeling Trickcited by 4
- 2024TracrBench: Generating Interpretability Testbeds with Large Language Modelscited by 4
- 2024Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learningcited by 4
- 2026Discovering Interpretable Algorithms by Decompiling Transformers to RASPcited by 4
- 2026Differentiable Faithfulness Alignment for Cross-Model Circuit Transfercited by 3
- 2024Neural Decompiling of Tracr Transformerscited by 2
- 2025BlackboxNLP-2025 MIB Shared Task: Exploring Ensemble Strategies for Circuit Localization Methodscited by 2
- 2026Scalable Circuit Learning for Interpreting Large Language Modelscited by 2
- 2026MechRL: Reinforcement Learning Agents Perform Circuit Discovery for Mechanistic Interpretabilitycited by 1
50 most cited of 51 All 51 (JSON)
Built on by
Not cited yet.