# A Simple Model for Subject Behavior in Subjective Experiments

arXiv: https://arxiv.org/abs/2004.02067 · HTML: https://mlsanity.com/paper/2004.02067/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2020.

## Built on by (most cited first, 15 of 15)

- [Subjective Assessment of High Dynamic Range Videos Under Different Ambient Conditions](https://mlsanity.com/paper/2209.10005/) (2022)
- [Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems](https://mlsanity.com/paper/2307.12975/) (2023)
- [Telepresence Video Quality Assessment](https://mlsanity.com/paper/2207.09956/) (2022)
- [HDR-ChipQA: No-Reference Quality Assessment on High Dynamic Range Videos](https://mlsanity.com/paper/2304.13156/) (2023)
- [Subjective Quality Assessment of Compressed Tone-Mapped High Dynamic Range Videos](https://mlsanity.com/paper/2403.15061/) (2024)
- [CHUG: Crowdsourced User-Generated HDR Video Quality Dataset](https://mlsanity.com/paper/2510.09879/) (2025)
- [Encoder-Quantization-Motion-based Video Quality Metrics](https://mlsanity.com/paper/2404.06620/) (2024)
- [Seeing Beyond 8bits: Subjective and Objective Quality Assessment of HDR-UGC Videos](https://mlsanity.com/paper/2603.00938/) (2026)
- [Learning with Noisy Low-Cost MOS for Image Quality Assessment via Dual-Bias Calibration](https://mlsanity.com/paper/2311.15846/) (2023)
- [Perceptual Constancy Constrained Single Opinion Score Calibration for Image Quality Assessment](https://mlsanity.com/paper/2404.19595/) (2024)
- [Beyond MOS: Subjective Image Quality Score Preprocessing Method Based on Perceptual Similarity](https://mlsanity.com/paper/2404.19666/) (2024)
- [Satellite Streaming Video QoE Prediction: A Real-World Subjective Database and Network-Level Prediction Models](https://mlsanity.com/paper/2410.13952/) (2024)
- [Subjective and Objective Quality Assessment of Banding Artifacts on Compressed Videos](https://mlsanity.com/paper/2508.08700/) (2025)
- [Adversarial Stress Testing of Outlier Detection in Subjective Image Quality Assessment](https://mlsanity.com/paper/2509.06554/) (2025)
- [GameScope: A Multi-Attribute, Multi-Codec Benchmark Dataset for Gaming Video Quality Assessment](https://mlsanity.com/paper/2605.01272/) (2026)

## Abstract (arXiv)

In a subjective experiment to evaluate the perceptual audiovisual quality of multimedia and television services, raw opinion scores collected from test subjects are often noisy and unreliable. To produce the final mean opinion scores (MOS), recommendations such as ITU-R BT.500, ITU-T P.910 and ITU-T P.913 standardize post-test screening procedures to clean up the raw opinion scores, using techniques such as subject outlier rejection and bias removal. In this paper, we analyze the prior standardized techniques to demonstrate their weaknesses. As an alternative, we propose a simple model to account for two of the most dominant behaviors of subject inaccuracy: bias and inconsistency. We further show that this model can also effectively deal with inattentive subjects that give random scores. We propose to use maximum likelihood estimation to jointly solve the model parameters, and present two numeric solvers: the first based on the Newton-Raphson method, and the second based on an alternating projection (AP). We show that the AP solver generalizes the ITU-T P.913 post-test screening procedure by weighing a subject's contribution to the true quality score by her consistency (thus, the quality scores estimated can be interpreted as bias-subtracted consistency-weighted MOS). We compare the proposed methods with the standardized techniques using real datasets and synthetic simulations, and demonstrate that the proposed methods are the most valuable when the test conditions are challenging (for example, crowdsourcing and cross-lab studies), offering advantages such as better model-data fit, tighter confidence intervals, better robustness against subject outliers, the absence of hard coded parameters and thresholds, and auxiliary information on test subjects. The code for this work is open-sourced at https://github.com/Netflix/sureal.