# A crowdsourcing approach to video quality assessment

arXiv: https://arxiv.org/abs/2204.06784 · HTML: https://mlsanity.com/paper/2204.06784/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2022.

## Built on by (most cited first, 8 of 8)

- [NTIRE 2025 Challenge on Video Quality Enhancement for Video Conferencing: Datasets, Methods and Results](https://mlsanity.com/paper/2505.18988/) (2025)
- [What Matters in Practical Learned Image Compression](https://mlsanity.com/paper/2605.05148/) (2026)
- [LSTM-based Video Quality Prediction Accounting for Temporal Distortions in Videoconferencing Calls](https://mlsanity.com/paper/2303.12761/) (2023)
- [Full Reference Video Quality Assessment for Machine Learning-Based Video Codecs](https://mlsanity.com/paper/2309.00769/) (2023)
- [ICME 2025 Grand Challenge on Video Super-Resolution for Video Conferencing](https://mlsanity.com/paper/2506.12269/) (2025)
- [MLVC: Multi-platform Learned Video Codec for Real-World Deployment](https://mlsanity.com/paper/2606.28027/) (2026)
- [Ensuring Reliable Participation in Subjective Video Quality Tests Across Platforms](https://mlsanity.com/paper/2509.20001/) (2025)
- [A Camera-Native Talking-Head Video Dataset for Various Computer Vision Tasks](https://mlsanity.com/paper/2603.26763/) (2026)

## Abstract (arXiv)

The gold standard for measuring video quality is the subjective test, and the most prevalent is the ITU-T Rec. P.910, a lab-based subjective standard in use for the past two decades. However, in practice using ITU-T Rec. P.910 is slow, expensive, and requires a lab, which all create barriers to usage. As a result, most research papers that need to measure video quality don't use ITU-T Rec. P.910 but rather metrics that are not well correlated to subjective opinion. We provide an open-source extension of ITU-T Rec. P.910 based on crowdsourcing principles which address the speed, usage cost, and barrier to usage issues. We implement Absolute Category Rating (ACR), ACR with hidden reference (ACR-HR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR). It includes rater, environment, hardware, and network qualifications, as well as gold and trapping questions to ensure quality. We have validated that the implementation is both accurate and highly reproducible compared to existing ITU-T Rec. P.910 lab studies.