# Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness

arXiv: https://arxiv.org/abs/2305.19101 · HTML: https://mlsanity.com/paper/2305.19101/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [The Riemannian Geometry of Deep Generative Models](https://mlsanity.com/paper/1711.08014/) (2017)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Knowledge Transfer with Jacobian Matching](https://mlsanity.com/paper/1803.00443/) (2018)
- [Fairwashing Explanations with Off-Manifold Detergent](https://mlsanity.com/paper/2007.09969/) (2020)
- [Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations](https://mlsanity.com/paper/2103.01890/) (2021)
- [Do Input Gradients Highlight Discriminative Features?](https://mlsanity.com/paper/2102.12781/) (2021)
- [Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?](https://mlsanity.com/paper/1910.08640/) (2019)
- [Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance](https://mlsanity.com/paper/2208.08664/) (2022)
- [Towards Understanding the Generative Capability of Adversarially Robust Classifiers](https://mlsanity.com/paper/2108.09093/) (2021)
- [The Manifold Hypothesis for Gradient-Based Explanations](https://mlsanity.com/paper/2206.07387/) (2022)
- [On the Benefits of Models with Perceptually-Aligned Gradients](https://mlsanity.com/paper/2005.01499/) (2020)
- [Class-Conditioned Transformation for Enhanced Robust Image Classification](https://mlsanity.com/paper/2303.15409/) (2023)

## Built on by (most cited first, 20 of 20)

- [Manifold Preserving Guided Diffusion](https://mlsanity.com/paper/2311.16424/) (2023)
- [Manifold Integrated Gradients: Riemannian Geometry for Feature Attribution](https://mlsanity.com/paper/2405.09800/) (2024)
- [Adversarial Examples Are Not Bugs, They Are Superposition](https://mlsanity.com/paper/2508.17456/) (2025)
- [CLIPAG: Towards Generator-Free Text-to-Image Generation](https://mlsanity.com/paper/2306.16805/) (2023)
- [Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models](https://mlsanity.com/paper/2404.13752/) (2024)
- [Adversarial Attacks Leverage Interference Between Features in Superposition](https://mlsanity.com/paper/2510.11709/) (2025)
- [Indirect Gradient Matching for Adversarial Robust Distillation](https://mlsanity.com/paper/2312.03286/) (2023)
- [Characterizing Model Robustness via Natural Input Gradients](https://mlsanity.com/paper/2409.20139/) (2024)
- [Characterizing Data Point Vulnerability via Average-Case Robustness](https://mlsanity.com/paper/2307.13885/) (2023)
- [Geometric Remove-and-Retrain (GOAR): Coordinate-Invariant eXplainable AI Assessment](https://mlsanity.com/paper/2407.12401/) (2024)
- [Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners](https://mlsanity.com/paper/2505.14042/) (2025)
- [Latent Guidance in Diffusion Models for Perceptual Evaluations](https://mlsanity.com/paper/2506.00327/) (2025)
- [Informative Post-Hoc Explanations Only Exist for Simple Functions](https://mlsanity.com/paper/2508.11441/) (2025)
- [Counterfactual Explanations on Robust Perceptual Geodesics](https://mlsanity.com/paper/2601.18678/) (2026)
- [Riemannian Metric Matching for Scalable Geometric Modeling of Distributions](https://mlsanity.com/paper/2606.14334/) (2026)
- [Generative Model Inversion Through the Lens of the Manifold Hypothesis](https://mlsanity.com/paper/2509.20177/) (2025)
- [Towards White Box Deep Learning](https://mlsanity.com/paper/2403.09863/) (2024)
- [Pulling Back the Curtain on Deep Networks](https://mlsanity.com/paper/2507.22832/) (2025)
- [Let the Carrier Carry the Attack: Preserving the Subject in Adversarial Image Generation](https://mlsanity.com/paper/2609.39723/) (2026)
- [Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders](https://mlsanity.com/paper/2610.11938/) (2026)

## Abstract (arXiv)

One of the remarkable properties of robust computer vision models is that their input-gradients are often aligned with human perception, referred to in the literature as perceptually-aligned gradients (PAGs). Despite only being trained for classification, PAGs cause robust models to have rudimentary generative capabilities, including image generation, denoising, and in-painting. However, the underlying mechanisms behind these phenomena remain unknown. In this work, we provide a first explanation of PAGs via off-manifold robustness, which states that models must be more robust off- the data manifold than they are on-manifold. We first demonstrate theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifold robustness, and confirm the same empirically for robust models trained via gradient norm regularization, randomized smoothing, and adversarial training with projected gradient descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-manifold robustness correlates well with perceptual alignment. Finally, based on the levels of on- and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak robustness, bayes-aligned robustness, and excessive robustness. Code is available at urlhttps://github.com/tml-tuebingen/pags.