# Recover Subjective Quality Scores from Noisy Measurements

arXiv: https://arxiv.org/abs/1611.01715 · HTML: https://mlsanity.com/paper/1611.01715/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2016.

## Built on by (most cited first, 8 of 8)

- [Video Quality Assessment: A Comprehensive Survey](https://mlsanity.com/paper/2412.04508/) (2024)
- [Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems](https://mlsanity.com/paper/2307.12975/) (2023)
- [Study of Subjective and Objective Quality Assessment of Mobile Cloud Gaming Videos](https://mlsanity.com/paper/2305.17260/) (2023)
- [Rethinking Crowdsourcing Annotation: Partial Annotation with Salient Labels for Multi-Label Image Classification](https://mlsanity.com/paper/2109.02688/) (2021)
- [Telepresence Video Quality Assessment](https://mlsanity.com/paper/2207.09956/) (2022)
- [Full Reference Video Quality Assessment for Machine Learning-Based Video Codecs](https://mlsanity.com/paper/2309.00769/) (2023)
- [Subjective and Objective Analysis of Indian Social Media Video Quality](https://mlsanity.com/paper/2401.02794/) (2024)
- [Satellite Streaming Video QoE Prediction: A Real-World Subjective Database and Network-Level Prediction Models](https://mlsanity.com/paper/2410.13952/) (2024)

## Abstract (arXiv)

Simple quality metrics such as PSNR are known to not correlate well with subjective quality when tested across a wide spectrum of video content or quality regime. Recently, efforts have been made in designing objective quality metrics trained on subjective data (e.g. VMAF), demonstrating better correlation with video quality perceived by human. Clearly, the accuracy of such a metric heavily depends on the quality of the subjective data that it is trained on. In this paper, we propose a new approach to recover subjective quality scores from noisy raw measurements, using maximum likelihood estimation, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together. We also derive closed-from expression for the confidence interval of each estimate. Compared to previous methods which partially exploit the subjective information, our approach is able to exploit the information in full, yielding tighter confidence interval and better handling of outliers without the need for z-scoring or subject rejection. It also handles missing data more gracefully. Finally, as side information, it provides interesting insights on the test subjects and video contents.