mlsanityData to 7 Oct 2026

Mechanistic Interpretability for AI Safety -- A Review

arXiv 2404.14082 ↗First posted 22 Apr 2024cs.AIcited by 415

Not on the map. Why?

Cited by 35 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding AI systems' inner workings is critical for ensuring value alignment and safety. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and representations learned by neural networks into human-understandable algorithms and concepts to provide a granular, causal understanding. We establish foundational concepts such as features encoding knowledge within neural activations and hypotheses about their representation and computation. We survey methodologies for causally dissecting model behaviors and assess the relevance of mechanistic interpretability to AI safety. We examine benefits in understanding, control, alignment, and risks such as capability gains and dual-use concerns. We investigate challenges surrounding scalability, automation, and comprehensive interpretation. We advocate for clarifying concepts, setting standards, and scaling techniques to handle complex models and behaviors and expand to domains such as vision and reinforcement learning. Mechanistic interpretability could help prevent catastrophic outcomes as AI systems become more powerful and inscrutable.

Key facts. Cited by 415 papers in the map (data through 7 Oct 2026). Builds on 202 of its 287 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

202 of 287 references.

References · 202

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
38 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  4. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  9. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  13. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  14. 2022Mass-Editing Memory in a Transformercited by 976
  15. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  16. 2022In-context Learning and Induction Headscited by 930
  17. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  18. 2022Toy Models of Superpositioncited by 928
  19. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  20. 2023Steering Language Models With Activation Engineeringcited by 824
  21. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  22. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  23. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  24. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  25. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  26. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  27. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  28. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  29. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  30. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  31. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  32. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  33. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  34. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  35. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  36. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  37. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  38. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417

50 most cited of 202 All 202 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 57; 2025: 188; 2026: 170.

415 papers.

Sort:

By area:

  1. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  2. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  3. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  4. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  5. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  6. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  7. 2025Agentic Large Language Models, a surveycited by 105
  8. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  9. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  10. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  11. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
  12. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
38 more
  1. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  3. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 54
  4. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  5. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  6. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 46
  7. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  8. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 39
  9. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 36
  10. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  11. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  12. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additionscited by 32
  13. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  14. 2025Mitigating Overthinking in Large Reasoning Models via Manifold Steeringcited by 32
  15. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  16. 2025Characterizing AI Agents for Alignment and Governancecited by 30
  17. 2025Superposition Yields Robust Neural Scalingcited by 30
  18. 2025Mechanistic understanding and validation of large AI models with SemanticLenscited by 29
  19. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  20. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  21. 2024Mechanistic?cited by 25
  22. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  23. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  24. 2025Mechanistic Interpretability of Emotion Inference in Large Language Modelscited by 24
  25. 2025Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionscited by 23
  26. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 23
  27. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 22
  28. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  29. 2024Evaluating Creative Short Story Generation in Humans and Large Language Modelscited by 21
  30. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identificationcited by 21
  31. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  32. 2026Sycophancy Hides Linearly in the Attention Headscited by 21
  33. 2024A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcutscited by 19
  34. 2025Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesiscited by 19
  35. 2025How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencodingcited by 19
  36. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  37. 2025The Rogue Scalpel: Activation Steering Compromises LLM Safetycited by 18
  38. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17

50 most cited of 415 All 415 (JSON)