# Identifying and Mitigating Flaws of Deep Perceptual Similarity Metrics

arXiv: https://arxiv.org/abs/2207.02512 · HTML: https://mlsanity.com/paper/2207.02512/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 6 Jul 2022.

## Builds on (most cited first, 16 of 16)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Beyond the Pixel-Wise Loss for Topology-Aware Delineation](https://mlsanity.com/paper/1712.02190/) (2017)
- [Perceptual Generative Adversarial Networks for Small Object Detection](https://mlsanity.com/paper/1706.05274/) (2017)
- [E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles](https://mlsanity.com/paper/1906.03973/) (2019)
- [Do better ImageNet classifiers assess perceptual similarity better?](https://mlsanity.com/paper/2203.04946/) (2022)
- [Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution](https://mlsanity.com/paper/1806.05764/) (2018)
- [An Unsupervised Information-Theoretic Perceptual Quality Metric](https://mlsanity.com/paper/2006.06752/) (2020)
- [Pretraining Image Encoders without Reconstruction via Feature Prediction Loss](https://mlsanity.com/paper/2003.07441/) (2020)

## Built on by (most cited first, 4 of 4)

- [LipSim: A Provably Robust Perceptual Similarity Metric](https://mlsanity.com/paper/2310.18274/) (2023)
- [Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics](https://mlsanity.com/paper/2502.11725/) (2025)
- [Counterfactual Explanations on Robust Perceptual Geodesics](https://mlsanity.com/paper/2601.18678/) (2026)
- [Deep Perceptual Similarity is Adaptable to Ambiguous Contexts](https://mlsanity.com/paper/2304.02265/) (2023)

## Abstract (arXiv)

Measuring the similarity of images is a fundamental problem to computer vision for which no universal solution exists. While simple metrics such as the pixel-wise L2-norm have been shown to have significant flaws, they remain popular. One group of recent state-of-the-art metrics that mitigates some of those flaws are Deep Perceptual Similarity (DPS) metrics, where the similarity is evaluated as the distance in the deep features of neural networks. However, DPS metrics themselves have been less thoroughly examined for their benefits and, especially, their flaws. This work investigates the most common DPS metric, where deep features are compared by spatial position, along with metrics comparing the averaged and sorted deep features. The metrics are analyzed in-depth to understand the strengths and weaknesses of the metrics by using images designed specifically to challenge them. This work contributes with new insights into the flaws of DPS, and further suggests improvements to the metrics. An implementation of this work is available online: https://github.com/guspih/deep_perceptual_similarity_analysis/