# A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations

arXiv: https://arxiv.org/abs/1805.07039 · HTML: https://mlsanity.com/paper/1805.07039/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2018.

## Built on by (most cited first, 30 of 60)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Impossibility Theorems for Feature Attribution](https://mlsanity.com/paper/2212.11870/) (2022)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation](https://mlsanity.com/paper/2212.04629/) (2022)
- [Framework for Evaluating Faithfulness of Local Explanations](https://mlsanity.com/paper/2202.00734/) (2022)
- [Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks](https://mlsanity.com/paper/2107.11400/) (2021)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Neural Response Interpretation through the Lens of Critical Pathways](https://mlsanity.com/paper/2103.16886/) (2021)
- [Don't trust your eyes: on the (un)reliability of feature visualizations](https://mlsanity.com/paper/2306.04719/) (2023)
- [What Makes a Good Explanation?: A Harmonized View of Properties of Explanations](https://mlsanity.com/paper/2211.05667/) (2022)
- [Do Explanations Explain? Model Knows Best](https://mlsanity.com/paper/2203.02269/) (2022)
- [Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information](https://mlsanity.com/paper/2110.01471/) (2021)
- [Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics](https://mlsanity.com/paper/2409.16756/) (2024)
- [ExSum: From Local Explanations to Model Understanding](https://mlsanity.com/paper/2205.00130/) (2022)
- [A Fine-grained Interpretability Evaluation Benchmark for Neural NLP](https://mlsanity.com/paper/2205.11097/) (2022)
- [On the Relationship Between Explanation and Prediction: A Causal View](https://mlsanity.com/paper/2212.06925/) (2022)
- [Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods](https://mlsanity.com/paper/2212.06872/) (2022)
- [The Solvability of Interpretability Evaluation Metrics](https://mlsanity.com/paper/2205.08696/) (2022)
- [Explainable AI for Bioinformatics: Methods, Tools, and Applications](https://mlsanity.com/paper/2212.13261/) (2022)
- [Understanding and Unifying Fourteen Attribution Methods with Taylor Interactions](https://mlsanity.com/paper/2303.01506/) (2023)
- [Explaining COVID-19 and Thoracic Pathology Model Predictions by Identifying Informative Input Features](https://mlsanity.com/paper/2104.00411/) (2021)
- [Pixel-Level Explanation of Multiple Instance Learning Models in Biomedical Single Cell Images](https://mlsanity.com/paper/2303.08632/) (2023)
- [Contrastive Counterfactual Visual Explanations With Overdetermination](https://mlsanity.com/paper/2106.14556/) (2021)
- [Provably Better Explanations with Optimized Aggregation of Feature Attributions](https://mlsanity.com/paper/2406.05090/) (2024)
- [Uncertainty Propagation in XAI: A Comparison of Analytical and Empirical Estimators](https://mlsanity.com/paper/2504.03736/) (2025)
- [Believe The HiPe: Hierarchical Perturbation for Fast, Robust, and Model-Agnostic Saliency Mapping](https://mlsanity.com/paper/2103.05108/) (2021)
- [A Functional Information Perspective on Model Interpretation](https://mlsanity.com/paper/2206.05700/) (2022)
- [From Heatmaps to Structural Explanations of Image Classifiers](https://mlsanity.com/paper/2109.06365/) (2021)
- [Mutual Information Preserving Back-propagation: Learn to Invert for Faithful Attribution](https://mlsanity.com/paper/2104.06629/) (2021)

## Abstract (arXiv)

Backpropagation-based visualizations have been proposed to interpret convolutional neural networks (CNNs), however a theory is missing to justify their behaviors: Guided backpropagation (GBP) and deconvolutional network (DeconvNet) generate more human-interpretable but less class-sensitive visualizations than saliency map. Motivated by this, we develop a theoretical explanation revealing that GBP and DeconvNet are essentially doing (partial) image recovery which is unrelated to the network decisions. Specifically, our analysis shows that the backward ReLU introduced by GBP and DeconvNet, and the local connections in CNNs are the two main causes of compelling visualizations. Extensive experiments are provided that support the theoretical analysis.