# SAM-IQA: Can Segment Anything Boost Image Quality Assessment?

arXiv: https://arxiv.org/abs/2307.04455 · HTML: https://mlsanity.com/paper/2307.04455/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Noise2Noise: Learning Image Restoration without Clean Data](https://mlsanity.com/paper/1803.04189/) (2018)
- [Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network](https://mlsanity.com/paper/1907.02665/) (2019)
- [Inpaint Anything: Segment Anything Meets Image Inpainting](https://mlsanity.com/paper/2304.06790/) (2023)
- [Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment](https://mlsanity.com/paper/1612.01697/) (2016)
- [PieAPP: Perceptual Image-Error Assessment through Pairwise Preference](https://mlsanity.com/paper/1806.02067/) (2018)
- [MetaIQA: Deep Meta-learning for No-Reference Image Quality Assessment](https://mlsanity.com/paper/2004.05508/) (2020)
- [NBNet: Noise Basis Learning for Image Denoising with Subspace Projection](https://mlsanity.com/paper/2012.15028/) (2020)
- [Perceptual Quality Prediction on Authentically Distorted Images Using a Bag of Features Approach](https://mlsanity.com/paper/1609.04757/) (2016)
- [Ghost-free High Dynamic Range Imaging with Context-aware Transformer](https://mlsanity.com/paper/2208.05114/) (2022)
- [Norm-in-Norm Loss with Faster Convergence and Better Performance for Image Quality Assessment](https://mlsanity.com/paper/2008.03889/) (2020)
- [Perceptual Image Quality Assessment with Transformers](https://mlsanity.com/paper/2104.14730/) (2021)
- [ADNet: Attention-guided Deformable Convolutional Network for High Dynamic Range Imaging](https://mlsanity.com/paper/2105.10697/) (2021)
- [Deep Constrained Least Squares for Blind Image Super-Resolution](https://mlsanity.com/paper/2202.07508/) (2022)
- [Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment](https://mlsanity.com/paper/2108.07948/) (2021)
- [NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results](https://mlsanity.com/paper/2106.01439/) (2021)
- [Blind Predicting Similar Quality Map for Image Quality Assessment](https://mlsanity.com/paper/1805.08493/) (2018)
- [Can SAM Boost Video Super-Resolution?](https://mlsanity.com/paper/2305.06524/) (2023)
- [DIPNet: Efficiency Distillation and Iterative Pruning for Image Super-Resolution](https://mlsanity.com/paper/2304.07018/) (2023)
- [Content-Variant Reference Image Quality Assessment via Knowledge Distillation](https://mlsanity.com/paper/2202.13123/) (2022)
- [Incorporating Semi-Supervised and Positive-Unlabeled Learning for Boosting Full Reference Image Quality Assessment](https://mlsanity.com/paper/2204.08763/) (2022)
- [D2C-SR: A Divergence to Convergence Approach for Real-World Image Super-Resolution](https://mlsanity.com/paper/2103.14373/) (2021)
- [Realistic Noise Synthesis with Diffusion Models](https://mlsanity.com/paper/2305.14022/) (2023)

## Built on by (most cited first, 1 of 1)

- [Video Quality Assessment: A Comprehensive Survey](https://mlsanity.com/paper/2412.04508/) (2024)

## Abstract (arXiv)

Image Quality Assessment (IQA) is a challenging task that requires training on massive datasets to achieve accurate predictions. However, due to the lack of IQA data, deep learning-based IQA methods typically rely on pre-trained networks trained on massive datasets as feature extractors to enhance their generalization ability, such as the ResNet network trained on ImageNet. In this paper, we utilize the encoder of Segment Anything, a recently proposed segmentation model trained on a massive dataset, for high-level semantic feature extraction. Most IQA methods are limited to extracting spatial-domain features, while frequency-domain features have been shown to better represent noise and blur. Therefore, we leverage both spatial-domain and frequency-domain features by applying Fourier and standard convolutions on the extracted features, respectively. Extensive experiments are conducted to demonstrate the effectiveness of all the proposed components, and results show that our approach outperforms the state-of-the-art (SOTA) in four representative datasets, both qualitatively and quantitatively. Our experiments confirm the powerful feature extraction capabilities of Segment Anything and highlight the value of combining spatial-domain and frequency-domain features in IQA tasks. Code: https://github.com/Hedlen/SAM-IQA