# Towards Robust Learning with Different Label Noise Distributions

arXiv: https://arxiv.org/abs/1912.08741 · HTML: https://mlsanity.com/paper/1912.08741/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2019.

## Built on by (most cited first, 14 of 14)

- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [PropMix: Hard Sample Filtering and Proportional MixUp for Learning with Noisy Labels](https://mlsanity.com/paper/2110.11809/) (2021)
- [Analysing the Noise Model Error for Realistic Noisy Label Data](https://mlsanity.com/paper/2101.09763/) (2021)
- [Addressing out-of-distribution label noise in webly-labelled data](https://mlsanity.com/paper/2110.13699/) (2021)
- [ScanMix: Learning from Severe Label Noise via Semantic Clustering and Semi-Supervised Learning](https://mlsanity.com/paper/2103.11395/) (2021)
- [Federated Noisy Client Learning](https://mlsanity.com/paper/2106.13239/) (2021)
- [Is your noise correction noisy? PLS: Robustness to label noise with two stage detection](https://mlsanity.com/paper/2210.04578/) (2022)
- [Embedding contrastive unsupervised features to cluster in- and out-of-distribution noise in corrupted image datasets](https://mlsanity.com/paper/2207.01573/) (2022)
- [Semi-supervised dry herbage mass estimation using automatic data and synthetic images](https://mlsanity.com/paper/2110.13719/) (2021)
- [Maximising the Utility of Validation Sets for Imbalanced Noisy-label Meta-learning](https://mlsanity.com/paper/2208.08132/) (2022)
- [Semantic-Syntactic Discrepancy in Images (SSDI): Learning Meaning and Order of Features from Natural Images](https://mlsanity.com/paper/2401.17515/) (2024)
- [A Robust Optimization Method for Label Noisy Datasets Based on Adaptive Threshold: Adaptive-k](https://mlsanity.com/paper/2203.14165/) (2022)
- [FLASH: Federated Learning Across Simultaneous Heterogeneities](https://mlsanity.com/paper/2402.08769/) (2024)
- [An accurate detection is not all you need to combat label noise in web-noisy datasets](https://mlsanity.com/paper/2407.05528/) (2024)

## Abstract (arXiv)

Noisy labels are an unavoidable consequence of labeling processes and detecting them is an important step towards preventing performance degradations in Convolutional Neural Networks. Discarding noisy labels avoids a harmful memorization, while the associated image content can still be exploited in a semi-supervised learning (SSL) setup. Clean samples are usually identified using the small loss trick, i.e. they exhibit a low loss. However, we show that different noise distributions make the application of this trick less straightforward and propose to continuously relabel all images to reveal a discriminative loss against multiple distributions. SSL is then applied twice, once to improve the clean-noisy detection and again for training the final model. We design an experimental setup based on ImageNet32/64 for better understanding the consequences of representation learning with differing label noise distributions and find that non-uniform out-of-distribution noise better resembles real-world noise and that in most cases intermediate features are not affected by label noise corruption. Experiments in CIFAR-10/100, ImageNet32/64 and WebVision (real-world noise) demonstrate that the proposed label noise Distribution Robust Pseudo-Labeling (DRPL) approach gives substantial improvements over recent state-of-the-art. Code is available at https://git.io/JJ0PV.