# Adversarial Stress Testing of Outlier Detection in Subjective Image Quality Assessment

arXiv: https://arxiv.org/abs/2509.06554 · HTML: https://mlsanity.com/paper/2509.06554/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 14 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Builds on (most cited first, 3 of 3)

- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [A Simple Model for Subject Behavior in Subjective Experiments](https://mlsanity.com/paper/2004.02067/) (2020)
- [Black-box Adversarial Attacks Against Image Quality Assessment Models](https://mlsanity.com/paper/2402.17533/) (2024)

## Abstract (arXiv)

In subjective image and video quality assessment, observers rate or compare selected stimuli. Before calculating mean opinion scores (MOSs), unreliable ratings should be identified and handled as outliers. Several outlier-detection methods are available, including standardized procedures, but their comparative performance is often evaluated using only specific types of synthetic outliers such as random clickers. Such tests do not necessarily reveal the worst-case behavior of these methods. To address this gap, we introduce and demonstrate a general empirical worst-case framework for outlier-detection methods, with proof-of-concept adversarial attack generators for both discrete absolute category and continuous visual analog scale ratings. The attacks use optimization algorithms to identify ratings that maximize the discrepancy between the resulting MOS estimates and the ground truth. We apply the proposed framework to several hard and soft outlier-detection methods and demonstrate substantial differences in their worst-case reconstruction performance under adversarial stress. We also propose several low-complexity outlier-detection methods that achieve excellent empirical worst-case performance.