mlsanityData to 14 Oct 2026

Optimal ablation for interpretability

arXiv 2409.09951 ↗First posted 16 Sep 2024cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Interpretability studies often involve tracing the flow of information through machine learning models to identify specific model components that perform relevant computations for tasks of interest. Prior work quantifies the importance of a model component on a particular task by measuring the impact of performing ablation on that component, or simulating model inference with the component disabled. We propose a new method, optimal ablation (OA), and show that OA-based component importance has theoretical and empirical advantages over measuring importance via other ablation methods. We also show that OA-based component importance can benefit several downstream interpretability tasks, including circuit discovery, localization of factual recall, and latent prediction.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 90 of its 119 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

90 of 119 references.

References · 90

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  9. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  12. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
38 more
  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  2. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  3. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  4. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  6. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  10. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  11. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
  12. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  13. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  14. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 373
  15. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  16. 2023Language Models Represent Space and Timecited by 316
  17. 2023In-Context Learning Creates Task Vectorscited by 312
  18. 2023Function Vectors in Large Language Modelscited by 308
  19. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  20. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 259
  21. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  22. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 250
  23. 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
  24. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  25. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  26. 2016Understanding Neural Networks through Representation Erasurecited by 224
  27. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
  28. 2024How to use and interpret activation patchingcited by 214
  29. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  30. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  31. 2023Localizing Model Behavior with Path Patchingcited by 195
  32. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  33. 2014Object Detectors Emerge in Deep Scene CNNscited by 179
  34. 2020Compositional Explanations of Neuronscited by 172
  35. 2019Feature relevance quantification in explainable AI: A causal problemcited by 167
  36. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  37. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  38. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 16.

25 papers.

Sort:

By area:

  1. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
  2. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 25
  3. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 9
  4. 2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
  5. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  6. 2025Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagationcited by 5
  7. 2025Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretabilitycited by 3
  8. 2026How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuitscited by 2
  9. 2026Explanation Multiplicity: Circuit-Level Interpretability Evidence Does Not Survive Defensible Analytic Variationcited by 2
  10. 2026The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQAcited by 2
  11. 2026Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decompositioncited by 2
  12. 2025SALVE: Sparse Autoencoder-Latent Vector Editing for Mechanistic Control of Neural Networkscited by 1
13 more
  1. 2026Through the Looking Glass: Directly Reading and Writing Transformerscited by 1
  2. 2025Sparse Neuron Ablation Triggers Catastrophic Collapse of the Language Core in Large Vision-Language Modelscited by 0
  3. 2025Sparse Concept Anchoring for Interpretable and Controllable Neural Representationscited by 0
  4. 2026Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMscited by 0
  5. 2026Feature Starvation as Geometric Instability in Sparse Autoencoderscited by 0
  6. 2026Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?cited by 0
  7. 2026Beyond receptive fields: sequence-pooled normalization can supply most of a sequence labeler's contextcited by 0
  8. 2026From Retrieval to Recognition:How Vision--Language Models Become OCR Specialistscited by 0
  9. 2026Rethinking Contextualization by Reinterpreting Attention Head Channelscited by 0
  10. 2026Rethinking Circuit Evaluation: Do Circuits Explain Model Errors?cited by 0
  11. 2026Every Ablation Is a Dose: Counterweights and the Semblance of Self-Repaircited by 0
  12. 2026FlyHack: Stress-Testing Mechanistic Claims in a Connectome-Constrained Visual Modelcited by 0
  13. 2026Target-Token Asymmetry Can Masquerade As Depth Specializationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MechaTerp-TRACE: A Novel Approach for Component Ablation Analysis in Language Models
  2. 2026Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability
  3. 2026Every Component Is a Lookup: One Linear Graph for Interaction, Composition and Attribution
  4. 2026Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language Explanations
  5. 2024Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference
  6. 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysis
  7. 2026Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models
  8. 2026Finding Interpretable Prompt-Specific Circuits in Language Models