# Eigen-Distortions of Hierarchical Representations

arXiv: https://arxiv.org/abs/1710.02266 · HTML: https://mlsanity.com/paper/1710.02266/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2017.

## Built on by (most cited first, 26 of 26)

- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop](https://mlsanity.com/paper/2210.00933/) (2022)
- [Quantifying Visual Image Quality: A Bayesian View](https://mlsanity.com/paper/2102.00195/) (2021)
- [From superposition to sparse codes: interpretable representations in neural networks](https://mlsanity.com/paper/2503.01824/) (2025)
- [On the relation between statistical learning and perceptual distances](https://mlsanity.com/paper/2106.04427/) (2021)
- [Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks](https://mlsanity.com/paper/2202.00838/) (2022)
- [DeepDC: Deep Distance Correlation as a Perceptual Image Quality Evaluator](https://mlsanity.com/paper/2211.04927/) (2022)
- [Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?](https://mlsanity.com/paper/2305.17023/) (2023)
- [Discriminating image representations with principal distortions](https://mlsanity.com/paper/2410.15433/) (2024)
- [Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4](https://mlsanity.com/paper/2203.06649/) (2022)
- [Deep Shape-Texture Statistics for Completely Blind Image Quality Evaluation](https://mlsanity.com/paper/2401.08107/) (2024)
- [Learning a distance measure from the information-estimation geometry of data](https://mlsanity.com/paper/2510.02514/) (2025)
- [Image Quality Assessment in the Modern Age](https://mlsanity.com/paper/2110.09699/) (2021)
- [MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding](https://mlsanity.com/paper/2512.12307/) (2025)
- [From Global to Granular: Revealing IQA Model Performance via Correlation Surface](https://mlsanity.com/paper/2601.21738/) (2026)
- [Layerwise complexity-matched learning yields an improved model of cortical area V2](https://mlsanity.com/paper/2312.11436/) (2023)
- [Explicitly Modeling Pre-Cortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness](https://mlsanity.com/paper/2409.16838/) (2024)
- [Uncovering the Computational Ingredients of Human-Like Representations in LLMs](https://mlsanity.com/paper/2510.01030/) (2025)
- [Face processing emerges from object-trained convolutional neural networks](https://mlsanity.com/paper/2405.18800/) (2024)
- [MAME: Multidimensional Adaptive Metamer Exploration with Human Perceptual Feedback](https://mlsanity.com/paper/2503.13212/) (2025)
- [Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariance](https://mlsanity.com/paper/2506.17040/) (2025)
- [Targeted perturbations reveal brain-like local coding axes in robustified, but not standard, ANN-based brain models](https://mlsanity.com/paper/2509.23333/) (2025)
- [Quasi Monte Carlo methods enable extremely low-dimensional deep generative models](https://mlsanity.com/paper/2601.18676/) (2026)
- [Beyond Activation Alignment: The Geometry of Neural Sensitivity](https://mlsanity.com/paper/2605.03222/) (2026)
- [Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders](https://mlsanity.com/paper/2609.39115/) (2026)
- [JLD: Perceptual Distance Through A Jacobian Lens](https://mlsanity.com/paper/2610.05967/) (2026)

## Abstract (arXiv)

We develop a method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans. Specifically, we utilize Fisher information to establish a model-derived prediction of sensitivity to local perturbations of an image. For a given image, we compute the eigenvectors of the Fisher information matrix with largest and smallest eigenvalues, corresponding to the model-predicted most- and least-noticeable image distortions, respectively. For human subjects, we then measure the amount of each distortion that can be reliably detected when added to the image. We use this method to test the ability of a variety of representations to mimic human perceptual sensitivity. We find that the early layers of VGG16, a deep neural network optimized for object recognition, provide a better match to human perception than later layers, and a better match than a 4-stage convolutional neural network (CNN) trained on a database of human ratings of distorted image quality. On the other hand, we find that simple models of early visual processing, incorporating one or more stages of local gain control, trained on the same database of distortion ratings, provide substantially better predictions of human sensitivity than either the CNN, or any combination of layers of VGG16.