# Massive Online Crowdsourced Study of Subjective and Objective Picture Quality

arXiv: https://arxiv.org/abs/1511.02919 · HTML: https://mlsanity.com/paper/1511.02919/

## Key facts

- Cited by 209 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2015.

## Built on by (most cited first, 30 of 209)

- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://mlsanity.com/paper/2211.04894/) (2022)
- [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](https://mlsanity.com/paper/2303.14968/) (2023)
- [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023)
- [TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment](https://mlsanity.com/paper/2308.03060/) (2023)
- [No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency](https://mlsanity.com/paper/2108.06858/) (2021)
- [Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models](https://mlsanity.com/paper/2311.06783/) (2023)
- [RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content](https://mlsanity.com/paper/2101.10955/) (2021)
- [Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception](https://mlsanity.com/paper/2108.08505/) (2021)
- [Re-IQA: Unsupervised Learning for Image Quality Assessment in the Wild](https://mlsanity.com/paper/2304.00451/) (2023)
- [Image Quality Assessment using Contrastive Learning](https://mlsanity.com/paper/2110.13266/) (2021)
- [Towards Open-ended Visual Quality Comparison](https://mlsanity.com/paper/2402.16641/) (2024)
- [VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank](https://mlsanity.com/paper/2505.14460/) (2025)
- [Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare](https://mlsanity.com/paper/2405.19298/) (2024)
- [Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment](https://mlsanity.com/paper/2210.05357/) (2022)
- [Data-Efficient Image Quality Assessment with Attention-Panel Decoder](https://mlsanity.com/paper/2304.04952/) (2023)
- [Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models](https://mlsanity.com/paper/2307.13981/) (2023)
- [Continual Learning for Blind Image Quality Assessment](https://mlsanity.com/paper/2102.09717/) (2021)
- [PyTorch Image Quality: Metrics for Image Quality Assessment](https://mlsanity.com/paper/2208.14818/) (2022)
- [Quality-Aware Image-Text Alignment for Opinion-Unaware Image Quality Assessment](https://mlsanity.com/paper/2403.11176/) (2024)
- [Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop](https://mlsanity.com/paper/2210.00933/) (2022)
- [Green Learning: Introduction, Examples and Outlook](https://mlsanity.com/paper/2210.00965/) (2022)
- [GAIA: Rethinking Action Quality Assessment for AI-Generated Videos](https://mlsanity.com/paper/2406.06087/) (2024)
- [NTIRE 2024 Challenge on Short-form UGC Video Quality Assessment: Methods and Results](https://mlsanity.com/paper/2404.11313/) (2024)
- [LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models](https://mlsanity.com/paper/2408.14008/) (2024)
- [Going the Extra Mile in Face Image Quality Assessment: A Novel Database and Model](https://mlsanity.com/paper/2207.04904/) (2022)
- [Exploring the Naturalness of AI-Generated Images](https://mlsanity.com/paper/2312.05476/) (2023)
- [Quantifying Visual Image Quality: A Bayesian View](https://mlsanity.com/paper/2102.00195/) (2021)
- [2AFC Prompting of Large Multimodal Models for Image Quality Assessment](https://mlsanity.com/paper/2402.01162/) (2024)
- [NTIRE 2024 Quality Assessment of AI-Generated Content Challenge](https://mlsanity.com/paper/2404.16687/) (2024)

## Abstract (arXiv)

Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Towards overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment subjective study. Our database consists of over 350000 opinion scores on 1162 images evaluated by over 7000 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective dataset. We also evaluate several top-performing blind Image Quality Assessment algorithms on it and present insights on how mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models.