# Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning

arXiv: https://arxiv.org/abs/2306.05497 · HTML: https://mlsanity.com/paper/2306.05497/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [Learning From Noisy Large-Scale Datasets With Minimal Supervision](https://mlsanity.com/paper/1701.01619/) (2017)
- [Masking: A New Perspective of Noisy Supervision](https://mlsanity.com/paper/1805.08193/) (2018)
- [Curriculum Loss: Robust Learning and Generalization against Label Corruption](https://mlsanity.com/paper/1905.10045/) (2019)
- [Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels](https://mlsanity.com/paper/2105.04522/) (2021)
- [Learning From Noisy Singly-labeled Data](https://mlsanity.com/paper/1712.04577/) (2017)
- [Training Convolutional Networks with Noisy Labels](https://mlsanity.com/paper/1406.2080/) (2014)
- [Asymmetric Loss Functions for Learning with Noisy Labels](https://mlsanity.com/paper/2106.03110/) (2021)
- [Investigating Why Contrastive Learning Benefits Robustness Against Label Noise](https://mlsanity.com/paper/2201.12498/) (2022)
- [Contrastive Learning Improves Model Robustness Under Label Noise](https://mlsanity.com/paper/2104.08984/) (2021)

## Built on by (most cited first, 3 of 3)

- [Generalizing Abstention for Noise-Robust Learning in Medical Image Segmentation](https://mlsanity.com/paper/2601.14039/) (2026)
- [Minimax Generalized Cross-Entropy](https://mlsanity.com/paper/2603.19874/) (2026)
- [Learning Sign Language Recognition under Label Noise: A Study of Noise-Robust Losses for Isolated and Continuous Settings](https://mlsanity.com/paper/2609.12885/) (2026)

## Abstract (arXiv)

Large annotated datasets inevitably contain noisy labels, which poses a major challenge for training deep neural networks as they easily memorize the labels. Noise-robust loss functions have emerged as a notable strategy to counteract this issue, but it remains challenging to create a robust loss function which is not susceptible to underfitting. Through a quantitative approach, this paper explores the limited overlap between the network output at initialization and regions of non-vanishing gradients of bounded loss functions in the initial learning phase. Using these insights, we address underfitting of several noise robust losses with a novel method denoted as logit bias, which adds a real number ε to the logit at the position of the correct class. The logit bias enables these losses to achieve state-of-the-art results, even on datasets like WebVision, consisting of over a million images from 1000 classes. In addition, we demonstrate that our method can be used to determine optimal parameters for several loss functions -- without having to train networks. Remarkably, our method determines the hyperparameters based on the number of classes, resulting in loss functions which require zero dataset or noise-dependent parameters.