# Learning Important Features Through Propagating Activation Differences

arXiv: https://arxiv.org/abs/1704.02685 · HTML: https://mlsanity.com/paper/1704.02685/

## Key facts

- Cited by 1,458 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2017.

## Built on by (most cited first, 30 of 1458)

- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving](https://mlsanity.com/paper/2310.01957/) (2023)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [CRAFT: Concept Recursive Activation FacTorization for Explainability](https://mlsanity.com/paper/2211.10154/) (2022)
- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures](https://mlsanity.com/paper/2207.01186/) (2022)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation](https://mlsanity.com/paper/2206.03208/) (2022)
- [Benchmarking and Survey of Explanation Methods for Black Box Models](https://mlsanity.com/paper/2102.13076/) (2021)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [FastSHAP: Real-Time Shapley Value Estimation](https://mlsanity.com/paper/2107.07436/) (2021)
- [Explaining in Style: Training a GAN to explain a classifier in StyleSpace](https://mlsanity.com/paper/2104.13369/) (2021)
- [Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes](https://mlsanity.com/paper/2111.15000/) (2021)
- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond](https://mlsanity.com/paper/2103.10689/) (2021)
- [Algorithms to estimate Shapley value feature attributions](https://mlsanity.com/paper/2207.07605/) (2022)
- [A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation](https://mlsanity.com/paper/2306.07304/) (2023)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [A Consistent and Efficient Evaluation Strategy for Attribution Methods](https://mlsanity.com/paper/2202.00449/) (2022)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)
- [Trustworthy Graph Neural Networks: Aspects, Methods and Trends](https://mlsanity.com/paper/2205.07424/) (2022)
- [B-cos Networks: Alignment is All We Need for Interpretability](https://mlsanity.com/paper/2205.10268/) (2022)

## Abstract (arXiv)

The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Deep Learning Important FeaTures), a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. By optionally giving separate consideration to positive and negative contributions, DeepLIFT can also reveal dependencies which are missed by other approaches. Scores can be computed efficiently in a single backward pass. We apply DeepLIFT to models trained on MNIST and simulated genomic data, and show significant advantages over gradient-based methods. Video tutorial: http://goo.gl/qKb7pL, ICML slides: bit.ly/deeplifticmlslides, ICML talk: https://vimeo.com/238275076, code: http://goo.gl/RM8jvH.