# Perceptual Quality-based Model Training under Annotator Label Uncertainty

arXiv: https://arxiv.org/abs/2403.10190 · HTML: https://mlsanity.com/paper/2403.10190/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Introspective Learning : A Two-Stage Approach for Inference in Neural Networks](https://mlsanity.com/paper/2209.08425/) (2022)
- [UNIQUE: Unsupervised Image Quality Estimation](https://mlsanity.com/paper/1810.06631/) (2018)
- [MS-UNIQUE: Multi-model and Sharpness-weighted Unsupervised Image Quality Estimation](https://mlsanity.com/paper/1811.08947/) (2018)

## Built on by (most cited first, 2 of 2)

- [A Large-scale Benchmark on Geological Fault Delineation Models: Domain Shift, Training Dynamics, Generalizability, Evaluation and Inferential Behavior](https://mlsanity.com/paper/2505.08585/) (2025)
- [CRACKS: Crowdsourcing Resources for Analysis and Categorization of Key Subsurface faults](https://mlsanity.com/paper/2408.11185/) (2024)

## Abstract (arXiv)

Annotators exhibit disagreement during data labeling, which can be termed as annotator label uncertainty. Annotator label uncertainty manifests in variations of labeling quality. Training with a single low-quality annotation per sample induces model reliability degradations. In this work, we first examine the effects of annotator label uncertainty in terms of the model's generalizability and prediction uncertainty. We observe that the model's generalizability and prediction uncertainty degrade with the presence of low-quality noisy labels. Meanwhile, our evaluation of existing uncertainty estimation algorithms indicates their incapability in response to annotator label uncertainty. To mitigate performance degradation, prior methods show that training models with labels collected from multiple independent annotators can enhance generalizability. However, they require massive annotations. Hence, we introduce a novel perceptual quality-based model training framework to objectively generate multiple labels for model training to enhance reliability, while avoiding massive annotations. Specifically, we first select a subset of samples with low perceptual quality scores ranked by statistical regularities of visual signals. We then assign de-aggregated labels to each sample in this subset to obtain a training set with multiple labels. Our experiments and analysis demonstrate that training with the proposed framework alleviates the degradation of generalizability and prediction uncertainty caused by annotator label uncertainty.