# LipSim: A Provably Robust Perceptual Similarity Metric

arXiv: https://arxiv.org/abs/2310.18274 · HTML: https://mlsanity.com/paper/2310.18274/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers](https://mlsanity.com/paper/1906.04584/) (2019)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation](https://mlsanity.com/paper/1705.08475/) (2017)
- [(Certified!!) Adversarial Robustness for Free!](https://mlsanity.com/paper/2206.10550/) (2022)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity](https://mlsanity.com/paper/2203.05151/) (2022)
- [Generalizable Adversarial Training via Spectral Normalization](https://mlsanity.com/paper/1811.07457/) (2018)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [A Unified Algebraic Perspective on Lipschitz Neural Networks](https://mlsanity.com/paper/2303.03169/) (2023)
- [Mind the box: l1-APGD for sparse adversarial attacks on image classifiers](https://mlsanity.com/paper/2103.01208/) (2021)
- [Direct Parameterization of Lipschitz-Bounded Deep Networks](https://mlsanity.com/paper/2301.11526/) (2023)
- [E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles](https://mlsanity.com/paper/1906.03973/) (2019)
- [Shift-tolerant Perceptual Similarity Metric](https://mlsanity.com/paper/2207.13686/) (2022)
- [Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks](https://mlsanity.com/paper/2208.03160/) (2022)
- [Theoretical evidence for adversarial robustness through randomization](https://mlsanity.com/paper/1902.01148/) (2019)
- [R-LPIPS: An Adversarially Robust Perceptual Similarity Metric](https://mlsanity.com/paper/2307.15157/) (2023)

## Built on by (most cited first, 15 of 15)

- [GenAI Arena: An Open Evaluation Platform for Generative Models](https://mlsanity.com/paper/2406.04485/) (2024)
- [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024)
- [Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics](https://mlsanity.com/paper/2502.11725/) (2025)
- [EMMA: Efficient Visual Alignment in Multi-Modal LLMs](https://mlsanity.com/paper/2410.02080/) (2024)
- [The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric](https://mlsanity.com/paper/2607.18237/) (2026)
- [Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics](https://mlsanity.com/paper/2412.10594/) (2024)
- [Robustness in Both Domains: CLIP Needs a Robust Text Encoder](https://mlsanity.com/paper/2506.03355/) (2025)
- [Counterfactual Explanations on Robust Perceptual Geodesics](https://mlsanity.com/paper/2601.18678/) (2026)
- [GUIDE: Enhancing Gradient Inversion Attacks in Federated Learning with Denoising Models](https://mlsanity.com/paper/2510.17621/) (2025)
- [Learning Better Certified Models from Empirically-Robust Teachers](https://mlsanity.com/paper/2602.02626/) (2026)
- [Stochastic BIQA: Median Randomized Smoothing for Certified Blind Image Quality Assessment](https://mlsanity.com/paper/2411.12575/) (2024)
- [Disentangling Safe and Unsafe Corruptions via Anisotropy and Locality](https://mlsanity.com/paper/2501.18098/) (2025)
- [FS-IQA: Certified Feature Smoothing for Robust Image Quality Assessment](https://mlsanity.com/paper/2508.05516/) (2025)
- [Real-Time Cooked Food Image Synthesis and Visual Cooking Progress Monitoring on Edge Devices](https://mlsanity.com/paper/2511.16965/) (2025)
- [The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue](https://mlsanity.com/paper/2606.01901/) (2026)

## Abstract (arXiv)

Recent years have seen growing interest in developing and applying perceptual similarity metrics. Research has shown the superiority of perceptual metrics over pixel-wise metrics in aligning with human perception and serving as a proxy for the human visual system. On the other hand, as perceptual metrics rely on neural networks, there is a growing concern regarding their resilience, given the established vulnerability of neural networks to adversarial attacks. It is indeed logical to infer that perceptual metrics may inherit both the strengths and shortcomings of neural networks. In this work, we demonstrate the vulnerability of state-of-the-art perceptual similarity metrics based on an ensemble of ViT-based feature extractors to adversarial attacks. We then propose a framework to train a robust perceptual similarity metric called LipSim (Lipschitz Similarity Metric) with provable guarantees. By leveraging 1-Lipschitz neural networks as the backbone, LipSim provides guarded areas around each data point and certificates for all perturbations within an ℓ2 ball. Finally, a comprehensive set of experiments shows the performance of LipSim in terms of natural and certified scores and on the image retrieval application. The code is available at https://github.com/SaraGhazanfari/LipSim.