# Active label cleaning for improved dataset quality under resource constraints

arXiv: https://arxiv.org/abs/2109.00574 · HTML: https://mlsanity.com/paper/2109.00574/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 64 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [Learning explanations that are hard to vary](https://mlsanity.com/paper/2009.00329/) (2020)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise](https://mlsanity.com/paper/2012.05458/) (2020)
- [Learning with Bounded Instance- and Label-dependent Label Noise](https://mlsanity.com/paper/1709.03768/) (2017)

## Built on by (most cited first, 30 of 44)

- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [Instance-Dependent Noisy Label Learning via Graphical Modelling](https://mlsanity.com/paper/2209.00906/) (2022)
- [CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators](https://mlsanity.com/paper/2210.06812/) (2022)
- [AQuA: A Benchmarking Tool for Label Quality Assessment](https://mlsanity.com/paper/2306.09467/) (2023)
- [Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed](https://mlsanity.com/paper/2205.14094/) (2022)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [Probably Approximately Correct Labels](https://mlsanity.com/paper/2506.10908/) (2025)
- [Efficient Adversarial Training With Data Pruning](https://mlsanity.com/paper/2207.00694/) (2022)
- [Active Label Correction for Semantic Segmentation with Foundation Models](https://mlsanity.com/paper/2403.10820/) (2024)
- [Label Noise-Robust Learning using a Confidence-Based Sieving Strategy](https://mlsanity.com/paper/2210.05330/) (2022)
- [Target specification bias, counterfactual prediction, and algorithmic fairness in healthcare](https://mlsanity.com/paper/2308.02081/) (2023)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [Sanitizing Manufacturing Dataset Labels Using Vision-Language Models](https://mlsanity.com/paper/2506.23465/) (2025)
- [ActiveLab: Active Learning with Re-Labeling by Multiple Annotators](https://mlsanity.com/paper/2301.11856/) (2023)
- [An Embedding is Worth a Thousand Noisy Labels](https://mlsanity.com/paper/2408.14358/) (2024)
- [Reconstructing short-lived particles using hypergraph representation learning](https://mlsanity.com/paper/2402.10149/) (2024)
- [Focused Active Learning for Histopathological Image Classification](https://mlsanity.com/paper/2404.04663/) (2024)
- [Active Label Refinement for Robust Training of Imbalanced Medical Image Classification Tasks in the Presence of High Label Noise](https://mlsanity.com/paper/2407.05973/) (2024)
- [Model-agnostic Selective Labeling with Provable Statistical Guarantees](https://mlsanity.com/paper/2510.14581/) (2025)
- [Bayesian Detector Combination for Object Detection with Crowdsourced Annotations](https://mlsanity.com/paper/2407.07958/) (2024)
- [MedSAGa: Few-shot Memory Efficient Medical Image Segmentation using Gradient Low-Rank Projection in SAM](https://mlsanity.com/paper/2407.15042/) (2024)
- [GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noise](https://mlsanity.com/paper/2411.05939/) (2024)
- [A2LC: Active and Automated Label Correction for Semantic Segmentation](https://mlsanity.com/paper/2506.11599/) (2025)
- [AutoVDC: Automated Vision Data Cleaning Using Vision-Language Models](https://mlsanity.com/paper/2507.12414/) (2025)
- [Managing Diabetic Retinopathy with Deep Learning: A Data Centric Overview](https://mlsanity.com/paper/2604.02448/) (2026)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [A Methodology for Auditable Trustworthiness Levels in AI Lifecycle Governance](https://mlsanity.com/paper/2607.16130/) (2026)
- [Tutorial on the development of AI models for medical image analysis](https://mlsanity.com/paper/2208.00766/) (2022)

## Abstract (arXiv)

Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models and have an often-overlooked confounding effect on the assessment of model performance. Nevertheless, employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare. This work advocates for a data-driven approach to prioritising samples for re-annotation - which we term "active label cleaning". We propose to rank instances according to estimated label correctness and labelling difficulty of each sample, and introduce a simulation framework to evaluate relabelling efficacy. Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection. Crucially, the proposed active label cleaning enables correcting labels up to 4 times more effectively than typical random selection in realistic conditions, making better use of experts' valuable time for improving dataset quality.