# Open-set Label Noise Can Improve Robustness Against Inherent Label Noise

arXiv: https://arxiv.org/abs/2106.10891 · HTML: https://mlsanity.com/paper/2106.10891/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 92 references that are in the map.
- First posted on arXiv 21 Jun 2021.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)

## Built on by (most cited first, 30 of 59)

- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [Detecting Corrupted Labels Without Training a Model to Predict](https://mlsanity.com/paper/2110.06283/) (2021)
- [OpenMix: Exploring Outlier Samples for Misclassification Detection](https://mlsanity.com/paper/2303.17093/) (2023)
- [Towards Robust Ranker for Text Retrieval](https://mlsanity.com/paper/2206.08063/) (2022)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features](https://mlsanity.com/paper/2202.01273/) (2022)
- [Mitigating Memorization of Noisy Labels via Regularization between Representations](https://mlsanity.com/paper/2110.09022/) (2021)
- [Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasets](https://mlsanity.com/paper/2206.08802/) (2022)
- [Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning](https://mlsanity.com/paper/2210.07805/) (2022)
- [On the Effectiveness of Out-of-Distribution Data in Self-Supervised Long-Tail Learning](https://mlsanity.com/paper/2306.04934/) (2023)
- [Identifying Hard Noise in Long-Tailed Sample Distribution](https://mlsanity.com/paper/2207.13378/) (2022)
- [Instance-Dependent Noisy Label Learning via Graphical Modelling](https://mlsanity.com/paper/2209.00906/) (2022)
- [Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributes](https://mlsanity.com/paper/2210.03175/) (2022)
- [Learning to Segment from Noisy Annotations: A Spatial Correction Approach](https://mlsanity.com/paper/2308.02498/) (2023)
- [Distributionally Robust Post-hoc Classifiers under Prior Shifts](https://mlsanity.com/paper/2309.08825/) (2023)
- [EAT: Towards Long-Tailed Out-of-Distribution Detection](https://mlsanity.com/paper/2312.08939/) (2023)
- [ReSup: Reliable Label Noise Suppression for Facial Expression Recognition](https://mlsanity.com/paper/2305.17895/) (2023)
- [NNG-Mix: Improving Semi-supervised Anomaly Detection with Pseudo-anomaly Generation](https://mlsanity.com/paper/2311.11961/) (2023)
- [Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection](https://mlsanity.com/paper/2405.19902/) (2024)
- [Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2410.02681/) (2024)
- [Unlocking the Power of Open Set : A New Perspective for Open-Set Noisy Label Learning](https://mlsanity.com/paper/2305.04203/) (2023)
- [Skeleton-Based Human Action Recognition with Noisy Labels](https://mlsanity.com/paper/2403.09975/) (2024)
- [Noisy Label Processing for Classification: A Survey](https://mlsanity.com/paper/2404.04159/) (2024)
- [AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction](https://mlsanity.com/paper/2412.18255/) (2024)
- [Mitigating Label Noise using Prompt-Based Hyperbolic Meta-Learning in Open-Set Domain Generalization](https://mlsanity.com/paper/2412.18342/) (2024)
- [Probabilistic Machine Learning for Noisy Labels in Earth Observation](https://mlsanity.com/paper/2504.03478/) (2025)
- [Bounded and Uniform Energy-based Out-of-distribution Detection for Graphs](https://mlsanity.com/paper/2504.13429/) (2025)
- [Fairness Improves Learning from Noisily Labeled Long-Tailed Data](https://mlsanity.com/paper/2303.12291/) (2023)
- [Exploring the Noise Robustness of Online Conformal Prediction](https://mlsanity.com/paper/2501.18363/) (2025)
- [When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification](https://mlsanity.com/paper/2505.16149/) (2025)

## Abstract (arXiv)

Learning with noisy labels is a practically challenging problem in weakly supervised learning. In the existing literature, open-set noises are always considered to be poisonous for generalization, similar to closed-set noises. In this paper, we empirically show that open-set noisy labels can be non-toxic and even benefit the robustness against inherent noisy labels. Inspired by the observations, we propose a simple yet effective regularization by introducing Open-set samples with Dynamic Noisy Labels (ODNL) into training. With ODNL, the extra capacity of the neural network can be largely consumed in a way that does not interfere with learning patterns from clean data. Through the lens of SGD noise, we show that the noises induced by our method are random-direction, conflict-free and biased, which may help the model converge to a flat minimum with superior stability and enforce the model to produce conservative predictions on Out-of-Distribution instances. Extensive experimental results on benchmark datasets with various types of noisy labels demonstrate that the proposed method not only enhances the performance of many existing robust algorithms but also achieves significant improvement on Out-of-Distribution detection tasks even in the label noise setting.