# Shift-tolerant Perceptual Similarity Metric

arXiv: https://arxiv.org/abs/2207.13686 · HTML: https://mlsanity.com/paper/2207.13686/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 27 Jul 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Why do deep convolutional networks generalize so poorly to small image transformations?](https://mlsanity.com/paper/1805.12177/) (2018)
- [Spatially Transformed Adversarial Examples](https://mlsanity.com/paper/1801.02612/) (2018)
- [EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis](https://mlsanity.com/paper/1612.07919/) (2016)
- [PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration](https://mlsanity.com/paper/2007.12142/) (2020)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [PieAPP: Perceptual Image-Error Assessment through Pairwise Preference](https://mlsanity.com/paper/1806.02067/) (2018)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [Delving Deeper into Anti-aliasing in ConvNets](https://mlsanity.com/paper/2008.09604/) (2020)
- [E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles](https://mlsanity.com/paper/1906.03973/) (2019)
- [A Loss Function for Generative Neural Networks Based on Watson's Perceptual Model](https://mlsanity.com/paper/2006.15057/) (2020)
- [An Unsupervised Information-Theoretic Perceptual Quality Metric](https://mlsanity.com/paper/2006.06752/) (2020)
- [Impact of Aliasing on Generalization in Deep Convolutional Networks](https://mlsanity.com/paper/2108.03489/) (2021)
- [Geodesics of learned representations](https://mlsanity.com/paper/1511.06394/) (2015)
- [What Does CNN Shift Invariance Look Like? A Visualization Study](https://mlsanity.com/paper/2011.04127/) (2020)

## Built on by (most cited first, 30 of 44)

- [Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution](https://mlsanity.com/paper/2501.11561/) (2025)
- [Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models](https://mlsanity.com/paper/2312.08962/) (2023)
- [Q-Insight: Understanding Image Quality via Visual Reinforcement Learning](https://mlsanity.com/paper/2503.22679/) (2025)
- [Enhancing Descriptive Image Quality Assessment with A Large-scale Multi-modal Dataset](https://mlsanity.com/paper/2405.18842/) (2024)
- [Image Super-Resolution with Text Prompt Diffusion](https://mlsanity.com/paper/2311.14282/) (2023)
- [Binarized Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2406.05723/) (2024)
- [LipSim: A Provably Robust Perceptual Similarity Metric](https://mlsanity.com/paper/2310.18274/) (2023)
- [Multiscale Sliced Wasserstein Distances as Perceptual Color Difference Measures](https://mlsanity.com/paper/2407.10181/) (2024)
- [VisualCritic: Making LMMs Perceive Visual Quality Like Humans](https://mlsanity.com/paper/2403.12806/) (2024)
- [BiM-VFI: Bidirectional Motion Field-Guided Frame Interpolation for Video with Non-uniform Motions](https://mlsanity.com/paper/2412.11365/) (2024)
- [DeepDC: Deep Distance Correlation as a Perceptual Image Quality Evaluator](https://mlsanity.com/paper/2211.04927/) (2022)
- [Attacking Perceptual Similarity Metrics](https://mlsanity.com/paper/2305.08840/) (2023)
- [Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics](https://mlsanity.com/paper/2502.11725/) (2025)
- [GS-QA: Comprehensive Quality Assessment Benchmark for Gaussian Splatting View Synthesis](https://mlsanity.com/paper/2502.13196/) (2025)
- [AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation](https://mlsanity.com/paper/2506.01061/) (2025)
- [My3DGen: A Scalable Personalized 3D Generative Model](https://mlsanity.com/paper/2307.05468/) (2023)
- [HarmonyIQA: Pioneering Benchmark and Model for Image Harmonization Quality Assessment](https://mlsanity.com/paper/2501.01116/) (2025)
- [Augmenting Perceptual Super-Resolution via Image Quality Predictors](https://mlsanity.com/paper/2504.18524/) (2025)
- [Misalignment-Robust Frequency Distribution Loss for Image Transformation](https://mlsanity.com/paper/2402.18192/) (2024)
- [Foundation Models Boost Low-Level Perceptual Similarity Metrics](https://mlsanity.com/paper/2409.07650/) (2024)
- [Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution](https://mlsanity.com/paper/2507.14367/) (2025)
- [Q-MambaIR: Accurate Quantized Mamba for Efficient Image Restoration](https://mlsanity.com/paper/2503.21970/) (2025)
- [UARE: A Unified Vision-Language Model for Image Quality Assessment, Restoration, and Enhancement](https://mlsanity.com/paper/2512.06750/) (2025)
- [XYScanNet: A State Space Model for Single Image Deblurring](https://mlsanity.com/paper/2412.10338/) (2024)
- [EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editing](https://mlsanity.com/paper/2603.14916/) (2026)
- [A New Dataset and Framework for Real-World Blurred Images Super-Resolution](https://mlsanity.com/paper/2407.14880/) (2024)
- [Quality Prediction of AI Generated Images and Videos: Emerging Trends and Opportunities](https://mlsanity.com/paper/2410.08534/) (2024)
- [The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric](https://mlsanity.com/paper/2607.18237/) (2026)
- [High-Quality Medical Image Generation from Free-hand Sketch](https://mlsanity.com/paper/2402.00353/) (2024)
- [Evaluating quality metrics through the lenses of psychophysical measurements of low-level vision](https://mlsanity.com/paper/2503.16264/) (2025)

## Abstract (arXiv)

Existing perceptual similarity metrics assume an image and its reference are well aligned. As a result, these metrics are often sensitive to a small alignment error that is imperceptible to the human eyes. This paper studies the effect of small misalignment, specifically a small shift between the input and reference image, on existing metrics, and accordingly develops a shift-tolerant similarity metric. This paper builds upon LPIPS, a widely used learned perceptual similarity metric, and explores architectural design considerations to make it robust against imperceptible misalignment. Specifically, we study a wide spectrum of neural network elements, such as anti-aliasing filtering, pooling, striding, padding, and skip connection, and discuss their roles in making a robust metric. Based on our studies, we develop a new deep neural network-based perceptual similarity metric. Our experiments show that our metric is tolerant to imperceptible shifts while being consistent with the human similarity judgment.