# Sample Selection with Uncertainty of Losses for Learning with Noisy Labels

arXiv: https://arxiv.org/abs/2106.00445 · HTML: https://mlsanity.com/paper/2106.00445/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 99 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 30 of 54)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)
- [PyOD: A Python Toolbox for Scalable Outlier Detection](https://mlsanity.com/paper/1901.01588/) (2019)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [Probabilistic End-to-end Noise Correction for Learning with Noisy Labels](https://mlsanity.com/paper/1903.07788/) (2019)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [Free Lunch for Few-shot Learning: Distribution Calibration](https://mlsanity.com/paper/2101.06395/) (2021)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)

## Built on by (most cited first, 30 of 79)

- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [To Smooth or Not? When Label Smoothing Meets Noisy Labels](https://mlsanity.com/paper/2106.04149/) (2021)
- [Mitigating Memorization of Noisy Labels by Clipping the Model Prediction](https://mlsanity.com/paper/2212.04055/) (2022)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Robust Training of Graph Neural Networks via Noise Governance](https://mlsanity.com/paper/2211.06614/) (2022)
- [BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency](https://mlsanity.com/paper/2303.12419/) (2023)
- [From Shortcuts to Triggers: Backdoor Defense with Denoised PoE](https://mlsanity.com/paper/2305.14910/) (2023)
- [IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models](https://mlsanity.com/paper/2310.10873/) (2023)
- [Towards Open Set Video Anomaly Detection](https://mlsanity.com/paper/2208.11113/) (2022)
- [Robust Reinforcement Learning from Corrupted Human Feedback](https://mlsanity.com/paper/2406.15568/) (2024)
- [PiCO+: Contrastive Label Disambiguation for Robust Partial Label Learning](https://mlsanity.com/paper/2201.08984/) (2022)
- [When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Method](https://mlsanity.com/paper/2211.10955/) (2022)
- [Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection](https://mlsanity.com/paper/2111.14932/) (2021)
- [On the Over-Memorization During Natural, Robust and Catastrophic Overfitting](https://mlsanity.com/paper/2310.08847/) (2023)
- [FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning](https://mlsanity.com/paper/2310.16412/) (2023)
- [Principled Multimodal Representation Learning](https://mlsanity.com/paper/2507.17343/) (2025)
- [Label-Noise Learning with Intrinsically Long-Tailed Data](https://mlsanity.com/paper/2208.09833/) (2022)
- [Instance-Dependent Noisy Label Learning via Graphical Modelling](https://mlsanity.com/paper/2209.00906/) (2022)
- [Late Stopping: Avoiding Confidently Learning from Mislabeled Examples](https://mlsanity.com/paper/2308.13862/) (2023)
- [Vision-Language Models are Strong Noisy Label Detectors](https://mlsanity.com/paper/2409.19696/) (2024)
- [Neighborhood-Regularized Self-Training for Learning with Few Labels](https://mlsanity.com/paper/2301.03726/) (2023)
- [Rethinking Precision of Pseudo Label: Test-Time Adaptation via Complementary Learning](https://mlsanity.com/paper/2301.06013/) (2023)
- [Uncertainty Quantification for Machine Learning in Healthcare: A Survey](https://mlsanity.com/paper/2505.02874/) (2025)
- [SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise](https://mlsanity.com/paper/2111.11288/) (2021)
- [Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalization](https://mlsanity.com/paper/2310.16391/) (2023)
- [Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds](https://mlsanity.com/paper/2306.03116/) (2023)
- [Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI](https://mlsanity.com/paper/2403.04551/) (2024)
- [Instance-dependent Label Distribution Estimation for Learning with Label Noise](https://mlsanity.com/paper/2212.08380/) (2022)
- [DyGen: Learning from Noisy Labels via Dynamics-Enhanced Generative Modeling](https://mlsanity.com/paper/2305.19395/) (2023)
- [Beyond Myopia: Learning from Positive and Unlabeled Data through Holistic Predictive Trends](https://mlsanity.com/paper/2310.04078/) (2023)

## Abstract (arXiv)

In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. There are actually two possibilities of a large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks "learn patterns first"; (b) it belongs to an underrepresented group of data and has not been selected yet. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the confidence intervals of losses derived from distribution-free concentration inequalities, but not losses themselves, are used for sample selection. In this way, we also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As a result, we can better explore underrepresented data that are correctly labeled but seem to be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.