mlsanityData to 7 Oct 2026

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

arXiv 2211.00593 ↗First posted 1 Nov 2022cs.LG cs.AI cs.CLcited by 1,130

Not on the map. Why?

Cited by 96 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Research in mechanistic interpretability seeks to explain behaviors of machine learning models in terms of their internal components. However, most previous work either focuses on simple behaviors in small models, or describes complicated behaviors in larger models with broad strokes. In this work, we bridge this gap by presenting an explanation for how GPT-2 small performs a natural language task called indirect object identification (IOI). Our explanation encompasses 26 attention heads grouped into 7 main classes, which we discovered using a combination of interpretability approaches relying on causal interventions. To our knowledge, this investigation is the largest end-to-end attempt at reverse-engineering a natural behavior "in the wild" in a language model. We evaluate the reliability of our explanation using three quantitative criteria--faithfulness, completeness and minimality. Though these criteria support our explanation, they also point to remaining gaps in our understanding. Our work provides evidence that a mechanistic understanding of large ML models is feasible, opening opportunities to scale our understanding to both larger models and more complex tasks.

Key facts. Cited by 1,130 papers in the map (data through 7 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 1 Nov 2022.

Builds on

18 of 31 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2019Attention is not Explanationcited by 916
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2021Causal Abstractions of Neural Networkscited by 401
  10. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  11. 2020Compositional Explanations of Neuronscited by 172
  12. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
6 more
  1. 2022Analyzing Transformers in Embedding Spacecited by 131
  2. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  3. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  4. 2021An Interpretability Illusion for BERTcited by 94
  5. 2022X-Risk Analysis for AI Researchcited by 60
  6. 2022A Literature Survey of Recent Advances in Chatbotscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 112; 2024: 251; 2025: 297; 2026: 470.

1,130 papers.

Sort:

By area:

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  4. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  7. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  10. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  11. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  12. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
38 more
  1. 2023Language Models Represent Space and Timecited by 306
  2. 2023Function Vectors in Large Language Modelscited by 292
  3. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  4. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  5. 2023Large Language Model Alignment: A Surveycited by 241
  6. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  8. 2023An Overview of Catastrophic AI Riskscited by 214
  9. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  10. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  11. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  12. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  13. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  14. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  15. 2024How to use and interpret activation patchingcited by 191
  16. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  17. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  18. 2023Localizing Model Behavior with Path Patchingcited by 182
  19. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  20. 2023PMET: Precise Model Editing in a Transformercited by 170
  21. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  22. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  23. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  24. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  25. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  26. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  27. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  28. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  29. 2023The Quantization Model of Neural Scalingcited by 137
  30. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  31. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  32. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 118
  33. 2024KAN 2.0: Kolmogorov-Arnold Networks Meet Sciencecited by 116
  34. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  35. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  36. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  37. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  38. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111

50 most cited of 1,130 All 1,130 (JSON)