Noise Correction on Subjective Datasets
Not on the map. Why?
Abstract (arXiv)
Incorporating every annotator's perspective is crucial for unbiased data modeling. Annotator fatigue and changing opinions over time can distort dataset annotations. To combat this, we propose to learn a more accurate representation of diverse opinions by utilizing multitask learning in conjunction with loss-based label correction. We show that using our novel formulation, we can cleanly separate agreeing and disagreeing annotations. Furthermore, this method provides a controllable way to encourage or discourage disagreement. We demonstrate that this modification can improve prediction performance in a single or multi-annotator setting. Lastly, we show that this method remains robust to additional label noise that is applied to subjective data.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 1 Nov 2023.
Builds on
16 of 33 references.
References · 16
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
- 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
- 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
- 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
- 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
- 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
- 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
- 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
4 more
- 2021Augmentation Strategies for Learning with Noisy Labelscited by 86
- 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
- 2018Crowd disagreement about medical images is informativecited by 13
- 2019Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Biascited by 11
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.