mlsanityData to 7 Oct 2026

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

arXiv 2103.14749 ↗First posted 26 Mar 2021stat.ML cs.AI cs.LGcited by 388

Not on the map. Why?

Abstract (arXiv)

We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results. Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set. Putative label errors are identified using confident learning algorithms and then human-validated via crowdsourcing (51% of the algorithmically-flagged candidates are indeed erroneously labeled, on average across the datasets). Traditionally, machine learning practitioners choose which model to deploy based on test accuracy - our findings advise caution here, proposing that judging models over correctly labeled test sets may be more useful, especially for noisy real-world datasets. Surprisingly, we find that lower capacity models may be practically more useful than higher capacity models in real-world datasets with high proportions of erroneously labeled data. For example, on ImageNet with corrected labels: ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6%. On CIFAR-10 with corrected labels: VGG-11 outperforms VGG-19 if the prevalence of originally mislabeled test examples increases by just 5%. Test set errors across the 10 datasets can be viewed at https://labelerrors.com and all label errors can be reproduced by https://github.com/cleanlab/label-errors.

Key facts. Cited by 388 papers in the map (data through 7 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 26 Mar 2021.

Builds on

26 of 55 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  5. 2017A Closer Look at Memorization in Deep Networkscited by 936
  6. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  7. 2015Image-based Recommendations on Styles and Substitutescited by 702
  8. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  9. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  10. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  11. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
  12. 2017A Neural Representation of Sketch Drawingscited by 311
14 more
  1. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  2. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  3. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  4. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  5. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  6. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
  7. 2020From ImageNet to Image Classification: Contextualizing Progress on Benchmarkscited by 90
  8. 2020Fair Classification with Group-Dependent Label Noisecited by 67
  9. 2019Understanding Generalization through Visualizationscited by 54
  10. 2017Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labelscited by 52
  11. 2020Improving Generalization by Controlling Label-Noise Information in Neural Network Weightscited by 35
  12. 2017Learning Deep Networks from Noisy Labels with Dropout Regularizationcited by 32
  13. 2020A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations?cited by 25
  14. 2016Loss factorization, weakly supervised learning and label noise robustnesscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 78; 2023: 73; 2024: 64; 2025: 66; 2026: 65.

388 papers.

Sort:

By area:

  1. 2023Towards Unbounded Machine Unlearningcited by 261
  2. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  3. 2021Partial success in closing the gap between human and machine visioncited by 185
  4. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 184
  5. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 162
  6. 2021The Benchmark Lotterycited by 133
  7. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  8. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  9. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  10. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  11. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  12. 2021Demystifying the Draft EU Artificial Intelligence Actcited by 82
38 more
  1. 2022Towards Adversarial Evaluations for Inexact Machine Unlearningcited by 82
  2. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  3. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 82
  4. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  5. 2023Efficient Benchmarking of Language Modelscited by 71
  6. 2023LAVA: Data Valuation without Pre-Specified Learning Algorithmscited by 69
  7. 2024When Do We Not Need Larger Vision Models?cited by 65
  8. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 60
  9. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 57
  10. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  11. 2023OpenDataVal: a Unified Benchmark for Data Valuationcited by 55
  12. 2022Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifierscited by 53
  13. 2023Synthetic Datasets for Autonomous Driving: A Surveycited by 48
  14. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  15. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  16. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  17. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  18. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  19. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  20. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 39
  21. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  22. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  23. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  24. 2024Are large language models superhuman chemists?cited by 34
  25. 2022Data-Centric Artificial Intelligencecited by 31
  26. 2022Green Learning: Introduction, Examples and Outlookcited by 30
  27. 2025Fluid Language Model Benchmarkingcited by 30
  28. 2024Rethinking Data Shapley for Data Selection Tasks: Misleads and Meritscited by 29
  29. 2022Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Featurescited by 27
  30. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  31. 2022Towards Federated Learning against Noisy Labels via Local Self-Regularizationcited by 26
  32. 2024Corrective Machine Unlearningcited by 26
  33. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  34. 2022CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learningcited by 24
  35. 2022Bugs in the Data: How ImageNet Misrepresents Biodiversitycited by 24
  36. 2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 24
  37. 2023Adaptive conformal classification with noisy labelscited by 24
  38. 2025On Benchmarking Human-Like Intelligence in Machinescited by 24

50 most cited of 388 All 388 (JSON)