# The Manifold Hypothesis for Gradient-Based Explanations

arXiv: https://arxiv.org/abs/2206.07387 · HTML: https://mlsanity.com/paper/2206.07387/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 73 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 30 of 47)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)
- [Explaining by Removing: A Unified Framework for Model Explanation](https://mlsanity.com/paper/2011.14878/) (2020)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Disentangling Adversarial Robustness and Generalization](https://mlsanity.com/paper/1812.00740/) (2018)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Fooling Neural Network Interpretations via Adversarial Model Manipulation](https://mlsanity.com/paper/1902.02041/) (2019)

## Built on by (most cited first, 16 of 16)

- [Manifold Preserving Guided Diffusion](https://mlsanity.com/paper/2311.16424/) (2023)
- [Manifold Integrated Gradients: Riemannian Geometry for Feature Attribution](https://mlsanity.com/paper/2405.09800/) (2024)
- [Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness](https://mlsanity.com/paper/2305.19101/) (2023)
- [Explainable Molecular Property Prediction: Aligning Chemical Concepts with Predictions via Language Models](https://mlsanity.com/paper/2405.16041/) (2024)
- [Geometric Remove-and-Retrain (GOAR): Coordinate-Invariant eXplainable AI Assessment](https://mlsanity.com/paper/2407.12401/) (2024)
- [Tangentially Aligned Integrated Gradients for User-Friendly Explanations](https://mlsanity.com/paper/2503.08240/) (2025)
- [EquiReg: Equivariance Regularized Diffusion for Inverse Problems](https://mlsanity.com/paper/2505.22973/) (2025)
- [Riemannian Integrated Gradients: A Geometric View of Explainable AI](https://mlsanity.com/paper/2503.00892/) (2025)
- [Measuring the (Un)Faithfulness of Concept-Based Explanations](https://mlsanity.com/paper/2504.10833/) (2025)
- [Latent Guidance in Diffusion Models for Perceptual Evaluations](https://mlsanity.com/paper/2506.00327/) (2025)
- [Manifold-Aligned Guided Integrated Gradients for Reliable Feature Attribution](https://mlsanity.com/paper/2605.02167/) (2026)
- [Generative Model Inversion Through the Lens of the Manifold Hypothesis](https://mlsanity.com/paper/2509.20177/) (2025)
- [Post-hoc and manifold explanations analysis of facial expression data based on deep learning](https://mlsanity.com/paper/2404.18352/) (2024)
- [Algebraic Adversarial Attacks on Explainability Models](https://mlsanity.com/paper/2503.12683/) (2025)
- [Graph-based Integrated Gradients for Explaining Graph Neural Networks](https://mlsanity.com/paper/2509.07648/) (2025)
- [Concept-based explanation of gene expression prediction from H&E images](https://mlsanity.com/paper/2608.16669/) (2026)

## Abstract (arXiv)

When do gradient-based explanation algorithms provide perceptually-aligned explanations? We propose a criterion: the feature attributions need to be aligned with the tangent space of the data manifold. To provide evidence for this hypothesis, we introduce a framework based on variational autoencoders that allows to estimate and generate image manifolds. Through experiments across a range of different datasets -- MNIST, EMNIST, CIFAR10, X-ray pneumonia and Diabetic Retinopathy detection -- we demonstrate that the more a feature attribution is aligned with the tangent space of the data, the more perceptually-aligned it tends to be. We then show that the attributions provided by popular post-hoc methods such as Integrated Gradients and SmoothGrad are more strongly aligned with the data manifold than the raw gradient. Adversarial training also improves the alignment of model gradients with the data manifold. As a consequence, we suggest that explanation algorithms should actively strive to align their explanations with the data manifold. This is an extended version of a CVPR Workshop paper. Code is available at https://github.com/tml-tuebingen/explanations-manifold.