# Discriminating image representations with principal distortions

arXiv: https://arxiv.org/abs/2410.15433 · HTML: https://mlsanity.com/paper/2410.15433/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Generalized Shape Metrics on Neural Representations](https://mlsanity.com/paper/2110.14739/) (2021)
- [Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future](https://mlsanity.com/paper/2001.07092/) (2020)
- [Neural population geometry: An approach for understanding biological and artificial neural networks](https://mlsanity.com/paper/2104.07059/) (2021)
- [Spatial-frequency channels, shape bias, and adversarial robustness](https://mlsanity.com/paper/2309.13190/) (2023)
- [Eigen-Distortions of Hierarchical Representations](https://mlsanity.com/paper/1710.02266/) (2017)
- [Representational dissimilarity metric spaces for stochastic neural networks](https://mlsanity.com/paper/2211.11665/) (2022)
- [Robustified ANNs Reveal Wormholes Between Human Category Percepts](https://mlsanity.com/paper/2308.06887/) (2023)
- [Most discriminative stimuli for functional cell type clustering](https://mlsanity.com/paper/2401.05342/) (2023)

## Built on by (most cited first, 8 of 8)

- [MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding](https://mlsanity.com/paper/2512.12307/) (2025)
- [Supervised Quadratic Feature Analysis: Information Geometry Approach for Dimensionality Reduction](https://mlsanity.com/paper/2502.00168/) (2025)
- [MAME: Multidimensional Adaptive Metamer Exploration with Human Perceptual Feedback](https://mlsanity.com/paper/2503.13212/) (2025)
- [Targeted perturbations reveal brain-like local coding axes in robustified, but not standard, ANN-based brain models](https://mlsanity.com/paper/2509.23333/) (2025)
- [Beyond Activation Alignment: The Geometry of Neural Sensitivity](https://mlsanity.com/paper/2605.03222/) (2026)
- [Perceived face similarity reflects inverse-generative and naturalistic discriminative objectives](https://mlsanity.com/paper/2605.12619/) (2026)
- [Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders](https://mlsanity.com/paper/2609.39115/) (2026)
- [JLD: Perceptual Distance Through A Jacobian Lens](https://mlsanity.com/paper/2610.05967/) (2026)

## Abstract (arXiv)

Image representations (artificial or biological) are often compared in terms of their global geometric structure; however, representations with similar global structure can have strikingly different local geometries. Here, we propose a framework for comparing a set of image representations in terms of their local geometries. We quantify the local geometry of a representation using the Fisher information matrix, a standard statistical tool for characterizing the sensitivity to local stimulus distortions, and use this as a substrate for a metric on the local geometry in the vicinity of a base image. This metric may then be used to optimally differentiate a set of models, by finding a pair of "principal distortions" that maximize the variance of the models under this metric. As an example, we use this framework to compare a set of simple models of the early visual system, identifying a novel set of image distortions that allow immediate comparison of the models by visual inspection. In a second example, we apply our method to a set of deep neural network models and reveal differences in the local geometry that arise due to architecture and training types. These examples demonstrate how our framework can be used to probe for informative differences in local sensitivities between complex models, and suggest how it could be used to compare model representations with human perception.