mlsanityData to 14 Oct 2026

Confident Learning: Estimating Uncertainty in Dataset Labels

arXiv 1911.00068 ↗First posted 31 Oct 2019stat.ML cs.LGcited by 413

Not on the map. Why?

Abstract (arXiv)

Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. Whereas numerous studies have developed these principles independently, here, we combine them, building on the assumption of a class-conditional noise process to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This results in a generalized CL which is provably consistent and experimentally performant. We present sufficient conditions where CL exactly finds label errors, and show CL performance exceeding seven recent competitive approaches for learning with noisy labels on the CIFAR dataset. Uniquely, the CL framework is not coupled to a specific data modality or model (e.g., we use CL to find several label errors in the presumed error-free MNIST dataset and improve sentiment classification on text data in Amazon Reviews). We also employ CL on ImageNet to quantify ontological class overlap (e.g., estimating 645 "missile" images are mislabeled as their parent class "projectile"), and moderately increase model accuracy (e.g., for ResNet) by cleaning data prior to training. These results are replicable using the open-source cleanlab release.

Key facts. Cited by 413 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 47; 2022: 60; 2023: 75; 2024: 71; 2025: 74; 2026: 86.

413 papers.

Sort:

By area:

  1. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  2. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 165
  3. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  4. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  5. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 107
  6. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 90
  7. 2021Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognitioncited by 89
  8. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 89
  9. 2022Towards Adversarial Evaluations for Inexact Machine Unlearningcited by 84
  10. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 80
  11. 2023Efficient Benchmarking of Language Modelscited by 71
  12. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
38 more
  1. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  2. 2021Clusterability as an Alternative to Anchor Points When Learning with Noisy Labelscited by 65
  3. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 61
  4. 2022Hyperbolic Contrastive Learning for Visual Representations beyond Objectscited by 61
  5. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 58
  6. 2022CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentationcited by 58
  7. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  8. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  9. 2021Interactive Label Cleaning with Example-based Explanationscited by 49
  10. 2023Synthetic Datasets for Autonomous Driving: A Surveycited by 48
  11. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
  12. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  13. 2021Active label cleaning for improved dataset quality under resource constraintscited by 44
  14. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 41
  15. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  16. 2021Faster Meta Update Strategy for Noise-Robust Deep Learningcited by 38
  17. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  18. 2023Making LLMs Worth Every Penny: Resource-Limited Text Classification in Bankingcited by 37
  19. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  20. 2021Discrete Representations Strengthen Vision Transformer Robustnesscited by 35
  21. 2022Identifiability of Label Noise Transition Matrixcited by 34
  22. 2024On Effects of Steering Latent Representation for Large Language Model Unlearningcited by 32
  23. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  24. 2024Corrective Machine Unlearningcited by 29
  25. 2022Uncertainty-Aware Learning Against Label Noise on Imbalanced Datasetscited by 28
  26. 2022Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Featurescited by 27
  27. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  28. 2023Memorization Through the Lens of Curvature of Loss Function Around Samplescited by 27
  29. 2023Understanding and Mitigating the Label Noise in Pre-training on Downstream Taskscited by 27
  30. 2023Unveiling Safety Vulnerabilities of Large Language Modelscited by 27
  31. 2024Automated Data Curation for Robust Language Model Fine-Tuningcited by 27
  32. 2021Mitigating Memorization of Noisy Labels via Regularization between Representationscited by 26
  33. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  34. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  35. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  36. 2021Superpixel-guided Iterative Learning from Noisy Labels for Medical Image Segmentationcited by 25
  37. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  38. 2021ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networkscited by 24

50 most cited of 413 All 413 (JSON)