Benchmarking noisy label detection methods
Not on the map. Why?
Abstract (arXiv)
Label noise is a common problem in real-world datasets, affecting both model training and validation. Clean data are essential for achieving strong performance and ensuring reliable evaluation. While various techniques have been proposed to detect noisy labels (or label errors), there is no clear consensus on optimal approaches. We perform a comprehensive benchmark of detection methods by decomposing them into three fundamental components: gathering strategy (in-sample vs out-of-sample), label disagreement measure, and aggregation method. This decomposition can be applied to many existing detection methods, and enables systematic comparison across diverse approaches. To fairly compare methods, we propose a unified benchmark task: detecting a fraction of training samples equal to the dataset's noise rate. We also introduce a novel metric: the false negative rate at this fixed operating point. Our evaluation spans vision and tabular datasets under both synthetic and real-world noise conditions. We identify that in-sample gathering using average probability aggregation combined with the logit margin as the label disagreement measure achieves the best results across most scenarios. Our findings provide practical guidance for designing new detection methods and selecting techniques for specific applications.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 17 Oct 2025.
Builds on
30 of 40 references.
References · 30
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
- 2017A Closer Look at Memorization in Deep Networkscited by 942
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
- 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
- 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
- 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
- 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
- 2017Snapshot Ensembles: Train 1, get M for freecited by 350
- 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 328
- 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
- 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
18 more
- 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
- 2017CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noisecited by 181
- 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
- 2021Jo-SRC: A Contrastive Approach for Combating Noisy Labelscited by 80
- 2022UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learningcited by 77
- 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 58
- 2021Co-learning: Learning from Noisy Labels with Self-supervisioncited by 51
- 2023Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labelscited by 20
- 2022CTRL: Clustering Training Losses for Label Error Detectioncited by 17
- 2023Late Stopping: Avoiding Confidently Learning from Mislabeled Examplescited by 17
- 2022Centrality and Consistency: Two-Stage Clean Samples Identification for Learning with Instance-Dependent Noisy Labelscited by 16
- 2024CLIPCleaner: Cleaning Noisy Labels with CLIPcited by 16
- 2024Vision-Language Models are Strong Noisy Label Detectorscited by 16
- 2022Robustness to Label Noise Depends on the Shape of the Noise Distribution in Feature Spacecited by 15
- 2021Clean or Annotate: How to Spend a Limited Data Collection Budgetcited by 10
- 2024Learning Discriminative Dynamics with Label Corruption for Noisy Label Detectioncited by 9
- 2025Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimizationcited by 6
- 2025ChronoSelect: Robust Learning with Noisy Labels via Dynamics Temporal Memorycited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.