# Image Quality Assessment for Perceptual Image Restoration: A New Dataset, Benchmark and Metric

arXiv: https://arxiv.org/abs/2011.15002 · HTML: https://mlsanity.com/paper/2011.15002/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 26 of 26)

- [Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models](https://mlsanity.com/paper/2312.08962/) (2023)
- [Masked Image Training for Generalizable Deep Image Denoising](https://mlsanity.com/paper/2303.13132/) (2023)
- [Perceptual Image Quality Assessment with Transformers](https://mlsanity.com/paper/2104.14730/) (2021)
- [Sparse Sampling Transformer with Uncertainty-Driven Ranking for Unified Removal of Raindrops and Rain Streaks](https://mlsanity.com/paper/2308.14153/) (2023)
- [RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models](https://mlsanity.com/paper/2407.18035/) (2024)
- [Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment](https://mlsanity.com/paper/2108.07948/) (2021)
- [NTIRE 2022 Challenge on Perceptual Image Quality Assessment](https://mlsanity.com/paper/2206.11695/) (2022)
- [Harnessing Diffusion-Yielded Score Priors for Image Restoration](https://mlsanity.com/paper/2507.20590/) (2025)
- [PyTorch Image Quality: Metrics for Image Quality Assessment](https://mlsanity.com/paper/2208.14818/) (2022)
- [Low-Res Leads the Way: Improving Generalization for Super-Resolution by Self-Supervised Learning](https://mlsanity.com/paper/2403.02601/) (2024)
- [NTIRE 2021 Challenge on Perceptual Image Quality Assessment](https://mlsanity.com/paper/2105.03072/) (2021)
- [Region-Adaptive Deformable Network for Image Quality Assessment](https://mlsanity.com/paper/2104.11599/) (2021)
- [Incorporating Semi-Supervised and Positive-Unlabeled Learning for Boosting Full Reference Image Quality Assessment](https://mlsanity.com/paper/2204.08763/) (2022)
- [Exploring Vulnerabilities of No-Reference Image Quality Assessment Models: A Query-Based Black-Box Method](https://mlsanity.com/paper/2401.05217/) (2024)
- [(ASNA) An Attention-based Siamese-Difference Neural Network with Surrogate Ranking Loss function for Perceptual Image Quality Assessment](https://mlsanity.com/paper/2105.02531/) (2021)
- [Diffusion Model Based Visual Compensation Guidance and Visual Difference Analysis for No-Reference Image Quality Assessment](https://mlsanity.com/paper/2402.14401/) (2024)
- [Conformer and Blind Noisy Students for Improved Image Quality Assessment](https://mlsanity.com/paper/2204.12819/) (2022)
- [Human Imperceptible Attacks and Applications to Improve Fairness](https://mlsanity.com/paper/2111.15603/) (2021)
- [Position: Agentic Systems Constitute a Key Component of Next-Generation Intelligent Image Processing](https://mlsanity.com/paper/2505.16007/) (2025)
- [Perception-oriented Bidirectional Attention Network for Image Super-resolution Quality Assessment](https://mlsanity.com/paper/2509.06442/) (2025)
- [The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speech](https://mlsanity.com/paper/2609.13150/) (2026)
- [Can No-reference features help in Full-reference image quality estimation?](https://mlsanity.com/paper/2203.00845/) (2022)
- [Multi-Scale Features and Parallel Transformers Based Image Quality Assessment](https://mlsanity.com/paper/2204.09779/) (2022)
- [Two-stage domain adapted training for better generalization in real-world image restoration and super-resolution](https://mlsanity.com/paper/2106.00504/) (2021)
- [AdaQual-Diff: Diffusion-Based Image Restoration via Adaptive Quality Prompting](https://mlsanity.com/paper/2504.12605/) (2025)
- [EvoIQA - Explaining Image Distortions with Evolved White-Box Logic](https://mlsanity.com/paper/2603.15887/) (2026)

## Abstract (arXiv)

Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent perceptual IR algorithms based on generative adversarial networks (GANs) have brought in significant improvement on visual performance, but also pose great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. We present two questions: Can existing IQA methods objectively evaluate recent IR algorithms? With the focus on beating current benchmarks, are we getting better IR algorithms? To answer the questions and promote the development of IQA methods, we contribute a large-scale IQA dataset, called Perceptual Image Processing ALgorithms (PIPAL) dataset. Especially, this dataset includes the results of GAN-based IR algorithms, which are missing in previous datasets. We collect more than 1.13 million human judgments to assign subjective scores for PIPAL images using the more reliable Elo system. Based on PIPAL, we present new benchmarks for both IQA and SR methods. Our results indicate that existing IQA methods cannot fairly evaluate GAN-based IR algorithms. While using appropriate evaluation methods is important, IQA methods should also be updated along with the development of IR algorithms. At last, we shed light on how to improve the IQA performance on GAN-based distortion. Inspired by the find that the existing IQA methods have an unsatisfactory performance on the GAN-based distortion partially because of their low tolerance to spatial misalignment, we propose to improve the performance of an IQA network on GAN-based distortion by explicitly considering this misalignment. We propose the Space Warping Difference Network, which includes the novel l_2 pooling layers and Space Warping Difference layers. Experiments demonstrate the effectiveness of the proposed method.