mlsanityData to 14 Oct 2026

Active label cleaning for improved dataset quality under resource constraints

arXiv 2109.00574 ↗First posted 1 Sep 2021cs.CV cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models and have an often-overlooked confounding effect on the assessment of model performance. Nevertheless, employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare. This work advocates for a data-driven approach to prioritising samples for re-annotation - which we term "active label cleaning". We propose to rank instances according to estimated label correctness and labelling difficulty of each sample, and introduce a simulation framework to evaluate relabelling efficacy. Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection. Crucially, the proposed active label cleaning enables correcting labels up to 4 times more effectively than typical random selection in realistic conditions, making better use of experts' valuable time for improving dataset quality.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

34 of 64 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  7. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  8. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  9. 2019When Does Label Smoothing Help?cited by 988
  10. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  11. 2017A Closer Look at Memorization in Deep Networkscited by 942
  12. 2017Deep Bayesian Active Learning with Image Datacited by 819
22 more
  1. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  3. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  4. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  5. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  6. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  7. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  8. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  9. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  10. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  11. 2020Are we done with ImageNet?cited by 333
  12. 2019Human uncertainty makes classification more robustcited by 235
  13. 2020Does label smoothing mitigate label noise?cited by 204
  14. 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
  15. 2020Learning explanations that are hard to varycited by 132
  16. 2019Learning From Noisy Labels By Regularized Estimation Of Annotator Confusioncited by 127
  17. 2020Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noisecited by 82
  18. 2017Learning with Bounded Instance- and Label-dependent Label Noisecited by 57
  19. 2017Learning From Noisy Singly-labeled Datacited by 50
  20. 2017Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathycited by 44
  21. 2019Exploring large scale public medical image datasetscited by 37
  22. 2020Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labelscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 7; 2024: 10; 2025: 9; 2026: 12.

44 papers.

Sort:

By area:

  1. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 46
  2. 2023A comprehensive survey on deep active learning in medical image analysiscited by 35
  3. 2022Instance-Dependent Noisy Label Learning via Graphical Modellingcited by 17
  4. 2022CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotatorscited by 16
  5. 2023AQuA: A Benchmarking Tool for Label Quality Assessmentcited by 16
  6. 2022Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbedcited by 12
  7. 2023Data Cleaning and Machine Learning: A Systematic Literature Reviewcited by 9
  8. 2025In the Picture: Medical Imaging Datasets, Artifacts, and their Living Reviewcited by 8
  9. 2025Probably Approximately Correct Labelscited by 8
  10. 2022Efficient Adversarial Training With Data Pruningcited by 7
  11. 2024Active Label Correction for Semantic Segmentation with Foundation Modelscited by 7
  12. 2022Label Noise-Robust Learning using a Confidence-Based Sieving Strategycited by 6
32 more
  1. 2023Target specification bias, counterfactual prediction, and algorithmic fairness in healthcarecited by 6
  2. 2024LEMoN: Label Error Detection using Multimodal Neighborscited by 6
  3. 2025Sanitizing Manufacturing Dataset Labels Using Vision-Language Modelscited by 6
  4. 2023ActiveLab: Active Learning with Re-Labeling by Multiple Annotatorscited by 5
  5. 2024An Embedding is Worth a Thousand Noisy Labelscited by 5
  6. 2024Reconstructing short-lived particles using hypergraph representation learningcited by 4
  7. 2024Focused Active Learning for Histopathological Image Classificationcited by 3
  8. 2024Active Label Refinement for Robust Training of Imbalanced Medical Image Classification Tasks in the Presence of High Label Noisecited by 3
  9. 2025Model-agnostic Selective Labeling with Provable Statistical Guaranteescited by 3
  10. 2024Bayesian Detector Combination for Object Detection with Crowdsourced Annotationscited by 2
  11. 2024MedSAGa: Few-shot Memory Efficient Medical Image Segmentation using Gradient Low-Rank Projection in SAMcited by 2
  12. 2024GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noisecited by 2
  13. 2025A2LC: Active and Automated Label Correction for Semantic Segmentationcited by 2
  14. 2025AutoVDC: Automated Vision Data Cleaning Using Vision-Language Modelscited by 1
  15. 2026Managing Diabetic Retinopathy with Deep Learning: A Data Centric Overviewcited by 1
  16. 2026To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systemscited by 1
  17. 2026A Methodology for Auditable Trustworthiness Levels in AI Lifecycle Governancecited by 1
  18. 2022Tutorial on the development of AI models for medical image analysiscited by 0
  19. 2023Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Programcited by 0
  20. 2023Sample selection with noise rate estimation in noise learning of medical image analysiscited by 0
  21. 2024MMIL: A novel algorithm for disease associated cell type discoverycited by 0
  22. 2025Bias-Aware Mislabeling Detection via Decoupled Confident Learningcited by 0
  23. 2025From Label Error Detection to Correction: A Modular Framework and Benchmark for Object Detection Datasetscited by 0
  24. 2026Confident Learning for Object Detection under Model Constraintscited by 0
  25. 2026Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Imagescited by 0
  26. 2026Weakly Supervised Patch Annotation for Improved Screening of Diabetic Retinopathycited by 0
  27. 2026Model-agnostic information transfer and fusion for classification with label noisecited by 0
  28. 2026Architecture-agnostic Lipschitz-constant Bayesian header and its application to resolve semantically proximal classification errors with vision transformerscited by 0
  29. 2026Novel Dynamic Batch-Sensitive Adam Optimiser for Vehicular Accident Injury Severity Predictioncited by 0
  30. 2026Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalancecited by 0
  31. 2026A Novel Method to Evaluate Models on Unreliable, Noisy and Inconsistent Labels: Adaptive Resolution Label Aggregation (ARLA)cited by 0
  32. 2026From internal representations to model improvement through prediction errorscited by 0