# Mitigating Memorization of Noisy Labels by Clipping the Model Prediction

arXiv: https://arxiv.org/abs/2212.04055 · HTML: https://mlsanity.com/paper/2212.04055/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 85 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 30 of 63)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)

## Built on by (most cited first, 30 of 40)

- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [Process Reward Model with Q-Value Rankings](https://mlsanity.com/paper/2410.11287/) (2024)
- [Sequence Length Independent Norm-Based Generalization Bounds for Transformers](https://mlsanity.com/paper/2310.13088/) (2023)
- [Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models](https://mlsanity.com/paper/2311.11202/) (2023)
- [Why is SAM Robust to Label Noise?](https://mlsanity.com/paper/2405.03676/) (2024)
- [Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond](https://mlsanity.com/paper/2408.11338/) (2024)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?](https://mlsanity.com/paper/2404.01775/) (2024)
- [Learning with Complementary Labels Revisited: The Selected-Completely-at-Random Setting Is More Practical](https://mlsanity.com/paper/2311.15502/) (2023)
- [SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness](https://mlsanity.com/paper/2403.08618/) (2024)
- [Slight Corruption in Pre-training Data Makes Better Diffusion Models](https://mlsanity.com/paper/2405.20494/) (2024)
- [Regularly Truncated M-estimators for Learning with Noisy Labels](https://mlsanity.com/paper/2309.00894/) (2023)
- [Tackling Noisy Labels with Network Parameter Additive Decomposition](https://mlsanity.com/paper/2403.13241/) (2024)
- [On the Learnability of Out-of-distribution Detection](https://mlsanity.com/paper/2404.04865/) (2024)
- [Learning from Noisy Labels via Conditional Distributionally Robust Optimization](https://mlsanity.com/paper/2411.17113/) (2024)
- [ε-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise](https://mlsanity.com/paper/2508.02387/) (2025)
- [Selective Learning: Towards Robust Calibration with Dynamic Regularization](https://mlsanity.com/paper/2402.08384/) (2024)
- [Exploring the Noise Robustness of Online Conformal Prediction](https://mlsanity.com/paper/2501.18363/) (2025)
- [Making Binary Classification from Multiple Unlabeled Datasets Almost Free of Supervision](https://mlsanity.com/paper/2306.07036/) (2023)
- [Optimized Gradient Clipping for Noisy Label Learning](https://mlsanity.com/paper/2412.08941/) (2024)
- [Improve Noise Tolerance of Robust Loss via Noise-Awareness](https://mlsanity.com/paper/2301.07306/) (2023)
- [Fuzzy-aware Loss for Source-free Domain Adaptation in Visual Emotion Recognition](https://mlsanity.com/paper/2501.15519/) (2025)
- [RCC-PFL: Robust Client Clustering under Noisy Labels in Personalized Federated Learning](https://mlsanity.com/paper/2503.19886/) (2025)
- [Robust Classification with Noisy Labels Based on Posterior Maximization](https://mlsanity.com/paper/2504.06805/) (2025)
- [FB-NLL: A Feature-Based Approach to Tackle Noisy Labels in Personalized Federated Learning](https://mlsanity.com/paper/2604.19729/) (2026)
- [Sharpness-Aware Minimization for Evolutionary Feature Construction in Regression](https://mlsanity.com/paper/2405.06869/) (2024)
- [Combating Semantic Contamination in Learning with Label Noise](https://mlsanity.com/paper/2412.11620/) (2024)
- [Batch Normalization Amplifies Memorization and Privacy Risks](https://mlsanity.com/paper/2605.24420/) (2026)
- [EchoAlign: Bridging Generative and Discriminative Learning under Noisy Labels](https://mlsanity.com/paper/2405.12969/) (2024)
- [Bayesian uncertainty-aware deep learning with noisy labels: Tackling annotation ambiguity in EEG seizure detection](https://mlsanity.com/paper/2410.19815/) (2024)

## Abstract (arXiv)

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.