mlsanityData to 14 Oct 2026

Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass

arXiv 2609.25285 ↗First posted 21 Sep 2026cs.AI cs.LGcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Finding circuits in language models usually means running many careful interventions. We try something simpler: treat attention as a routing map from one forward pass, keep a small set of routes that point toward the answer, and ask whether those routes actually matter. They often do. On induction and IOI (tasks where the "right" circuit is already known), ablating our extracted edges hurts the model much more than ablating a random set of the same size. We evaluate n=100 prompts per cell on GPT-2 Small, GPT-2 Medium, and Pythia-410M, with paired gap tests and bootstrap confidence intervals. The extract step costs one forward; a head-by-head patch sweep costs about two orders of magnitude more. We are not claiming a complete circuit atlas. We are claiming a cheap sketch that carries real causal signal on known tasks, with clear failure modes when it does not. Code and evaluation artifacts are at https://github.com/Aquinf03/live-circuit-routing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 9 of its 11 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

9 of 11 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 7

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2022In-context Learning and Induction Headscited by 964
  4. 2020Quantifying Attention Flow in Transformerscited by 775
  5. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  6. 2024How to use and interpret activation patchingcited by 214
  7. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Finding Interpretable Prompt-Specific Circuits in Language Models
  2. 2026Data-driven Circuit Discovery for Interpretability of Language Models
  3. 2026Architecture, Not Scale: Circuit Localization in Large Language Models
  4. 2025Position-aware Automatic Circuit Discovery
  5. 2026Circuit Claims Depend on What Is Extracted and How It Is Compared
  6. 2026How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits
  7. 2023Circuit Component Reuse Across Tasks in Transformer Language Models
  8. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition