mlsanityData to 7 Oct 2026

Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

arXiv 2309.16042 ↗First posted 27 Sep 2023cs.LG cs.AI cs.CLcited by 328

Not on the map. Why?

Cited by 35 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Mechanistic interpretability seeks to understand the internal mechanisms of machine learning models, where localization -- identifying the important model components -- is a key step. Activation patching, also known as causal tracing or interchange intervention, is a standard technique for this task (Vig et al., 2020), but the literature contains many variants with little consensus on the choice of hyperparameters or methodology. In this work, we systematically examine the impact of methodological details in activation patching, including evaluation metrics and corruption methods. In several settings of localization and circuit discovery in language models, we find that varying these hyperparameters could lead to disparate interpretability results. Backed by empirical observations, we give conceptual arguments for why certain metrics or methods may be preferred. Finally, we provide recommendations for the best practices of activation patching going forwards.

Key facts. Cited by 328 papers in the map (data through 7 Oct 2026). Builds on 44 of its 66 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

44 of 66 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
32 more
  1. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  3. 2021Causal Abstractions of Neural Networkscited by 401
  4. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  5. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  6. 2013Direct and Indirect Effectscited by 276
  7. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  8. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  9. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  10. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  11. 2023Localizing Model Behavior with Path Patchingcited by 182
  12. 2020Compositional Explanations of Neuronscited by 172
  13. 2019Feature relevance quantification in explainable AI: A causal problemcited by 164
  14. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  15. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  16. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  17. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  18. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  19. 2022Analyzing Transformers in Embedding Spacecited by 131
  20. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  21. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  22. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  23. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  24. 2023Explaining grokking through circuit efficiencycited by 105
  25. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  26. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  27. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  28. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  29. 2022Polysemanticity and Capacity in Neural Networkscited by 74
  30. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  31. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 33
  32. 2023NeuroSurgeon: A Toolkit for Subnetwork Analysiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 70; 2025: 75; 2026: 179.

328 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  3. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  4. 2024How to use and interpret activation patchingcited by 191
  5. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  6. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  7. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  8. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  9. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  10. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  11. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  12. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
38 more
  1. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  2. 2024Attention Heads of Large Language Models: A Surveycited by 65
  3. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  4. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  5. 2024How to think step-by-step: A mechanistic understanding of chain-of-thought reasoningcited by 53
  6. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  7. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  8. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  9. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  10. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 43
  11. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  12. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 37
  13. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  14. 2024Locating and Editing Factual Associations in Mambacited by 35
  15. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  16. 2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 33
  17. 2025(How) Do Language Models Track State?cited by 31
  18. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 30
  19. 2024Localizing Paragraph Memorization in Language Modelscited by 30
  20. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  21. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  22. 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
  23. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 28
  24. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  25. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 28
  26. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTcited by 27
  27. 2025Training Language Models to Explain Their Own Computationscited by 27
  28. 2024Finding Visual Task Vectorscited by 24
  29. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  30. 2025Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Modelscited by 23
  31. 2026How do LLMs Compute Verbal Confidencecited by 23
  32. 2024SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Modelscited by 22
  33. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  34. 2024Optimal ablation for interpretabilitycited by 21
  35. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  36. 2024Transformers Struggle to Learn to Searchcited by 19
  37. 2025How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencodingcited by 19
  38. 2024Transformer Circuit Faithfulness Metrics are not Robustcited by 18

50 most cited of 328 All 328 (JSON)