# Perceptual Image Quality Assessment with Transformers

arXiv: https://arxiv.org/abs/2104.14730 · HTML: https://mlsanity.com/paper/2104.14730/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 30 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index](https://mlsanity.com/paper/1308.3052/) (2013)
- [PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration](https://mlsanity.com/paper/2007.12142/) (2020)
- [PieAPP: Perceptual Image-Error Assessment through Pairwise Preference](https://mlsanity.com/paper/1806.02067/) (2018)
- [The 2018 PIRM Challenge on Perceptual Image Super-resolution](https://mlsanity.com/paper/1809.07517/) (2018)
- [Learning a No-Reference Quality Metric for Single-Image Super-Resolution](https://mlsanity.com/paper/1612.05890/) (2016)
- [Colorization Transformer](https://mlsanity.com/paper/2102.04432/) (2021)
- [Transformer for Image Quality Assessment](https://mlsanity.com/paper/2101.01097/) (2020)
- [Image Quality Assessment for Perceptual Image Restoration: A New Dataset, Benchmark and Metric](https://mlsanity.com/paper/2011.15002/) (2020)
- [Generative adversarial network-based image super-resolution using perceptual content losses](https://mlsanity.com/paper/1809.04783/) (2018)
- [Deep Learning-based Image Super-Resolution Considering Quantitative and Perceptual Quality](https://mlsanity.com/paper/1809.04789/) (2018)
- [Ambiguity of Objective Image Quality Metrics: A New Methodology for Performance Evaluation](https://mlsanity.com/paper/2101.07439/) (2021)

## Built on by (most cited first, 30 of 54)

- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment](https://mlsanity.com/paper/2308.03060/) (2023)
- [AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusion](https://mlsanity.com/paper/2310.10123/) (2023)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [Attentions Help CNNs See Better: Attention-based Hybrid Image Quality Assessment Network](https://mlsanity.com/paper/2204.10485/) (2022)
- [Joint Global and Local Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/2112.04487/) (2021)
- [NTIRE 2022 Challenge on Perceptual Image Quality Assessment](https://mlsanity.com/paper/2206.11695/) (2022)
- [ColorVideoVDP: A visual difference predictor for image, video and display distortions](https://mlsanity.com/paper/2401.11485/) (2024)
- [Quality Assessment in the Era of Large Models: A Survey](https://mlsanity.com/paper/2409.00031/) (2024)
- [Zoom-VQA: Patches, Frames and Clips Integration for Video Quality Assessment](https://mlsanity.com/paper/2304.06440/) (2023)
- [NTIRE 2021 Challenge on Perceptual Image Quality Assessment](https://mlsanity.com/paper/2105.03072/) (2021)
- [A Perceptual Quality Metric for Video Frame Interpolation](https://mlsanity.com/paper/2210.01879/) (2022)
- [Content-Variant Reference Image Quality Assessment via Knowledge Distillation](https://mlsanity.com/paper/2202.13123/) (2022)
- [SwinIQA: Learned Swin Distance for Compressed Image Quality Assessment](https://mlsanity.com/paper/2205.04264/) (2022)
- [Incorporating Semi-Supervised and Positive-Unlabeled Learning for Boosting Full Reference Image Quality Assessment](https://mlsanity.com/paper/2204.08763/) (2022)
- [Next Token Is Enough: Realistic Image Quality and Aesthetic Scoring with Multimodal Large Language Model](https://mlsanity.com/paper/2503.06141/) (2025)
- [MSTRIQ: No Reference Image Quality Assessment Based on Swin Transformer with Multi-Stage Fusion](https://mlsanity.com/paper/2205.10101/) (2022)
- [Vision-Language Consistency Guided Multi-modal Prompt Learning for Blind AI Generated Image Quality Assessment](https://mlsanity.com/paper/2406.16641/) (2024)
- [Progressive Training of A Two-Stage Framework for Video Restoration](https://mlsanity.com/paper/2204.09924/) (2022)
- [Conformer and Blind Noisy Students for Improved Image Quality Assessment](https://mlsanity.com/paper/2204.12819/) (2022)
- [LAR-IQA: A Lightweight, Accurate, and Robust No-Reference Image Quality Assessment Model](https://mlsanity.com/paper/2408.17057/) (2024)
- [AI-Generated Image Quality Assessment Based on Task-Specific Prompt and Multi-Granularity Similarity](https://mlsanity.com/paper/2411.16087/) (2024)
- [You Only Train Once: A Unified Framework for Both Full-Reference and No-Reference Image Quality Assessment](https://mlsanity.com/paper/2310.09560/) (2023)
- [Perception- and Fidelity-aware Reduced-Reference Super-Resolution Image Quality Assessment](https://mlsanity.com/paper/2405.09472/) (2024)
- [Impact of loss functions on the performance of a deep neural network designed to restore low-dose digital mammography](https://mlsanity.com/paper/2111.06890/) (2021)
- [Image Quality Assessment for Magnetic Resonance Imaging](https://mlsanity.com/paper/2203.07809/) (2022)
- [Image Quality Assessment with Gradient Siamese Network](https://mlsanity.com/paper/2208.04081/) (2022)
- [DCVQE: A Hierarchical Transformer for Video Quality Assessment](https://mlsanity.com/paper/2210.04377/) (2022)
- [Content-Diverse Comparisons improve IQA](https://mlsanity.com/paper/2211.05215/) (2022)
- [Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration](https://mlsanity.com/paper/2409.03455/) (2024)

## Abstract (arXiv)

In this paper, we propose an image quality transformer (IQT) that successfully applies a transformer architecture to a perceptual full-reference image quality assessment (IQA) task. Perceptual representation becomes more important in image quality assessment. In this context, we extract the perceptual feature representations from each of input images using a convolutional neural network (CNN) backbone. The extracted feature maps are fed into the transformer encoder and decoder in order to compare a reference and distorted images. Following an approach of the transformer-based vision models, we use extra learnable quality embedding and position embedding. The output of the transformer is passed to a prediction head in order to predict a final quality score. The experimental results show that our proposed model has an outstanding performance for the standard IQA datasets. For a large-scale IQA dataset containing output images of generative model, our model also shows the promising results. The proposed IQT was ranked first among 13 participants in the NTIRE 2021 perceptual image quality assessment challenge. Our work will be an opportunity to further expand the approach for the perceptual IQA task.