mlsanityData to 7 Oct 2026

A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models

arXiv 2407.02646 ↗First posted 2 Jul 2024cs.AI cs.CLcited by 113

Not on the map. Why?

Abstract (arXiv)

Mechanistic interpretability (MI) is an emerging sub-field of interpretability that seeks to understand a neural network model by reverse-engineering its internal computations. Recently, MI has garnered significant attention for interpreting transformer-based language models (LMs), resulting in many novel insights yet introducing new challenges. However, there has not been work that comprehensively reviews these insights and challenges, particularly as a guide for newcomers to this field. To fill this gap, we provide a comprehensive survey from a task-centric perspective, organizing the taxonomy of MI research around specific research questions or tasks. We outline the fundamental objects of study in MI, along with the techniques, evaluation methods, and key findings for each task in the taxonomy. In particular, we present a task-centric taxonomy as a roadmap for beginners to navigate the field by helping them quickly identify impactful problems in which they are most interested and leverage MI for their benefit. Finally, we discuss the current gaps in the field and suggest potential future directions for MI research.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 142 of its 196 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

142 of 196 references.

References · 142

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  8. 2016Layer Normalizationcited by 5,579
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  13. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  14. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  15. 2022Mass-Editing Memory in a Transformercited by 976
  16. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  17. 2022In-context Learning and Induction Headscited by 930
  18. 2022Toy Models of Superpositioncited by 928
  19. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  20. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  21. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  22. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  23. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  24. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  25. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  26. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  27. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  28. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  29. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  30. 2024Scaling and evaluating sparse autoencoderscited by 530
  31. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  32. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  33. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  34. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  35. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  36. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  37. 2021Causal Abstractions of Neural Networkscited by 401
  38. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396

50 most cited of 142 All 142 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 55; 2026: 47.

113 papers.

Sort:

By area:

  1. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  3. 2025Agentic Large Language Models, a surveycited by 105
  4. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  5. 2025Mitigating Overthinking in Large Reasoning Models via Manifold Steeringcited by 32
  6. 2024One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Modelscited by 30
  7. 2025Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionscited by 23
  8. 2026How do LLMs Compute Verbal Confidencecited by 23
  9. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 22
  10. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  11. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  12. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
38 more
  1. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
  2. 2025Large Language Models as Proxies for Theories of Human Linguistic Cognitioncited by 10
  3. 2025The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer trainingcited by 9
  4. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 9
  5. 2025An explainable transformer circuit for compositional generalizationcited by 8
  6. 2025VLAs are Confined yet Capable of Generalizing to Novel Instructionscited by 8
  7. 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gatescited by 8
  8. 2025A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinationscited by 8
  9. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 8
  10. 2025Distributional AGI Safetycited by 8
  11. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 7
  12. 2025CLUE: Conflict-guided Localization for LLM Unlearning Frameworkcited by 7
  13. 2025Large Language Models as Computable Approximations to Solomonoff Inductioncited by 6
  14. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  15. 2024Interpretability for Time Series Transformers using A Concept Bottleneck Frameworkcited by 4
  16. 2025Electronic Circuit Principles of Large Language Modelscited by 4
  17. 2025CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMscited by 4
  18. 2025LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgradescited by 4
  19. 2025Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Onescited by 4
  20. 2025Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effectscited by 4
  21. 2025Uncovering Grounding IDs: How External Cues Shape Multimodal Bindingcited by 4
  22. 2026Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMscited by 4
  23. 2025Deciphering Functions of Neurons in Vision-Language Modelscited by 3
  24. 2025Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semanticallycited by 3
  25. 2025ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behaviorcited by 3
  26. 2026Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoningcited by 3
  27. 2026Language Steering for Multilingual In-Context Learningcited by 3
  28. 2026CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphscited by 3
  29. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottleneckscited by 3
  30. 2026Automated Interpretability and Feature Discovery in Language Models with Agentscited by 3
  31. 2026Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Trainingcited by 3
  32. 2026Beyond the Black Box: Interpretability of Agentic AI Tool Usecited by 3
  33. 2024System 2 Reasoning Capabilities Are Nighcited by 2
  34. 2024Interpretable Next-token Prediction via the Generalized Induction Headcited by 2
  35. 2025Emergent Stack Representations in Modeling Counter Languages Using Transformerscited by 2
  36. 2025InfoCons: Identifying Interpretable Critical Concepts in Point Clouds via Information Theorycited by 2
  37. 2025FeatInv: Spatially resolved mapping from feature space to input space using conditional diffusion modelscited by 2
  38. 2025Task complexity shapes internal representations and robustness in neural networkscited by 2

50 most cited of 113 All 113 (JSON)