mlsanityData to 7 Oct 2026

AtP*: An efficient and scalable method for localizing LLM behaviour to components

arXiv 2403.00745 ↗First posted 1 Mar 2024cs.LG cs.CLcited by 100

Not on the map. Why?

Abstract (arXiv)

Activation Patching is a method of directly computing causal attributions of behavior to model components. However, applying it exhaustively requires a sweep with cost scaling linearly in the number of model components, which can be prohibitively expensive for SoTA Large Language Models (LLMs). We investigate Attribution Patching (AtP), a fast gradient-based approximation to Activation Patching and find two classes of failure modes of AtP which lead to significant false negatives. We propose a variant of AtP called AtP*, with two changes to address these failure modes while retaining scalability. We present the first systematic study of AtP and alternative methods for faster activation patching and show that AtP significantly outperforms all other investigated methods, with AtP* providing further significant improvement. Finally, we provide a method to bound the probability of remaining false negatives of AtP* estimates.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

30 of 48 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  12. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
18 more
  1. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  2. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  3. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  4. 2023Localizing Model Behavior with Path Patchingcited by 182
  5. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  6. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  7. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  8. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  9. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  10. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  11. 2023How do Language Models Bind Entities in Context?cited by 102
  12. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  13. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  14. 2023Improving Activation Steering in Language Models with Mean-Centringcited by 64
  15. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  16. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  17. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 33
  18. 2015PerforatedCNNs: Acceleration through Elimination of Redundant Convolutionscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 33; 2026: 37.

100 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  3. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  4. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  5. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  6. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  7. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  8. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  9. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  10. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  11. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 43
  12. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
38 more
  1. 2025Weight-sparse transformers have interpretable circuitscited by 40
  2. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  3. 2024AI and the Problem of Knowledge Collapsecited by 34
  4. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 33
  5. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  6. 2024Representational Analysis of Binding in Language Modelscited by 27
  7. 2024Mechanistic?cited by 25
  8. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  9. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  10. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  11. 2024Automatically Identifying Local and Global Circuits with Linear Computation Graphscited by 22
  12. 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 21
  13. 2025Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Videocited by 20
  14. 2024Transformer Circuit Faithfulness Metrics are not Robustcited by 18
  15. 2024Monotonic Representation of Numeric Properties in Language Modelscited by 15
  16. 2024Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Modelscited by 15
  17. 2025Position-aware Automatic Circuit Discoverycited by 14
  18. 2025Stochastic Parameter Decompositioncited by 14
  19. 2024Gumbel Counterfactual Generation From Language Modelscited by 13
  20. 2024Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Factscited by 13
  21. 2025Beyond Transcription: Mechanistic Interpretability in ASRcited by 12
  22. 2024Abrupt Learning in Transformers: A Case Study on Matrix Completioncited by 11
  23. 2025RomanLens: The Role Of Latent Romanization In Multilinguality In LLMscited by 11
  24. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  25. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  26. 2025RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patchingcited by 10
  27. 2025Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activationscited by 9
  28. 2025Can Interpretation Predict Behavior on Unseen Data?cited by 8
  29. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 7
  30. 2025Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compressioncited by 6
  31. 2025Circuit Stability Characterizes Language Model Generalizationcited by 6
  32. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  33. 2025Internal states before wait modulate reasoning patternscited by 6
  34. 2026Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuitscited by 6
  35. 2025Identifying Sparsely Active Circuits Through Local Loss Landscape Decompositioncited by 5
  36. 2026Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMscited by 5
  37. 2025On Relation-Specific Neurons in Large Language Modelscited by 4
  38. 2025Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagationcited by 4

50 most cited of 100 All 100 (JSON)