# Foundation Models Boost Low-Level Perceptual Similarity Metrics

arXiv: https://arxiv.org/abs/2409.07650 · HTML: https://mlsanity.com/paper/2409.07650/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration](https://mlsanity.com/paper/2007.12142/) (2020)
- [Shift-tolerant Perceptual Similarity Metric](https://mlsanity.com/paper/2207.13686/) (2022)
- [Do better ImageNet classifiers assess perceptual similarity better?](https://mlsanity.com/paper/2203.04946/) (2022)
- [R-LPIPS: An Adversarially Robust Perceptual Similarity Metric](https://mlsanity.com/paper/2307.15157/) (2023)
- [Attacking Perceptual Similarity Metrics](https://mlsanity.com/paper/2305.08840/) (2023)

## Built on by (most cited first, 6 of 6)

- [Do computer vision foundation models learn the low-level characteristics of the human visual system?](https://mlsanity.com/paper/2502.20256/) (2025)
- [Quality Prediction of AI Generated Images and Videos: Emerging Trends and Opportunities](https://mlsanity.com/paper/2410.08534/) (2024)
- [Latent Guidance in Diffusion Models for Perceptual Evaluations](https://mlsanity.com/paper/2506.00327/) (2025)
- [CAMP-VQA: Caption-Embedded Multimodal Perception for No-Reference Quality Assessment of Compressed Video](https://mlsanity.com/paper/2511.07290/) (2025)
- [Non-Aligned Reference Image Quality Assessment for Novel View Synthesis](https://mlsanity.com/paper/2511.08155/) (2025)
- [A unified Benchmark for Multi-Frame Image Restoration under Severe Refractive Warping](https://mlsanity.com/paper/2605.05079/) (2026)

## Abstract (arXiv)

For full-reference image quality assessment (FR-IQA) using deep-learning approaches, the perceptual similarity score between a distorted image and a reference image is typically computed as a distance measure between features extracted from a pretrained CNN or more recently, a Transformer network. Often, these intermediate features require further fine-tuning or processing with additional neural network layers to align the final similarity scores with human judgments. So far, most IQA models based on foundation models have primarily relied on the final layer or the embedding for the quality score estimation. In contrast, this work explores the potential of utilizing the intermediate features of these foundation models, which have largely been unexplored so far in the design of low-level perceptual similarity metrics. We demonstrate that the intermediate features are comparatively more effective. Moreover, without requiring any training, these metrics can outperform both traditional and state-of-the-art learned metrics by utilizing distance measures between the features.