# Mitigating Memorization of Noisy Labels via Regularization between Representations

arXiv: https://arxiv.org/abs/2110.09022 · HTML: https://mlsanity.com/paper/2110.09022/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Learning with Instance-Dependent Label Noise: A Sample Sieve Approach](https://mlsanity.com/paper/2010.02347/) (2020)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)

## Built on by (most cited first, 26 of 26)

- [Mitigating Memorization of Noisy Labels by Clipping the Model Prediction](https://mlsanity.com/paper/2212.04055/) (2022)
- [To Aggregate or Not? Learning with Separate Noisy Labels](https://mlsanity.com/paper/2206.07181/) (2022)
- [Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks](https://mlsanity.com/paper/2309.17002/) (2023)
- [Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models](https://mlsanity.com/paper/2311.11202/) (2023)
- [CTRL: Clustering Training Losses for Label Error Detection](https://mlsanity.com/paper/2208.08464/) (2022)
- [Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributes](https://mlsanity.com/paper/2210.03175/) (2022)
- [Fine-Grained Classification with Noisy Labels](https://mlsanity.com/paper/2303.02404/) (2023)
- [Mitigating Memorization In Language Models](https://mlsanity.com/paper/2410.02159/) (2024)
- [CroSel: Cross Selection of Confident Pseudo Labels for Partial-Label Learning](https://mlsanity.com/paper/2303.10365/) (2023)
- [Weakly-supervised Medical Image Segmentation with Gaze Annotations](https://mlsanity.com/paper/2407.07406/) (2024)
- [Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement](https://mlsanity.com/paper/2403.09101/) (2024)
- [Learning Discriminative Dynamics with Label Corruption for Noisy Label Detection](https://mlsanity.com/paper/2405.19902/) (2024)
- [Fine tuning Pre trained Models for Robustness Under Noisy Labels](https://mlsanity.com/paper/2310.17668/) (2023)
- [Impact of Noisy Supervision in Foundation Model Learning](https://mlsanity.com/paper/2403.06869/) (2024)
- [Skeleton-Based Human Action Recognition with Noisy Labels](https://mlsanity.com/paper/2403.09975/) (2024)
- [FedFixer: Mitigating Heterogeneous Label Noise in Federated Learning](https://mlsanity.com/paper/2403.16561/) (2024)
- [Noisy Label Processing for Classification: A Survey](https://mlsanity.com/paper/2404.04159/) (2024)
- [Mitigating Label Noise using Prompt-Based Hyperbolic Meta-Learning in Open-Set Domain Generalization](https://mlsanity.com/paper/2412.18342/) (2024)
- [Fairness Improves Learning from Noisily Labeled Long-Tailed Data](https://mlsanity.com/paper/2303.12291/) (2023)
- [An Empirical Study of Automated Mislabel Detection in Real World Vision Datasets](https://mlsanity.com/paper/2312.02200/) (2023)
- [Guiding Noisy Label Conditional Diffusion Models with Score-based Discriminator Correction](https://mlsanity.com/paper/2508.19581/) (2025)
- [Classifying Long-tailed and Label-noise Data via Disentangling and Unlearning](https://mlsanity.com/paper/2503.11414/) (2025)
- [Robust Classification with Noisy Labels Based on Posterior Maximization](https://mlsanity.com/paper/2504.06805/) (2025)
- [Can LLMs Learn to Reason Robustly under Noisy Supervision?](https://mlsanity.com/paper/2604.03993/) (2026)
- [Early Detection and Reduction of Memorisation for Domain Adaptation and Instruction Tuning](https://mlsanity.com/paper/2510.11372/) (2025)
- [EReLiFM: Evidential Reliability-Aware Residual Flow Meta-Learning for Open-Set Domain Generalization under Noisy Labels](https://mlsanity.com/paper/2510.12687/) (2025)

## Abstract (arXiv)

Designing robust loss functions is popular in learning with noisy labels while existing designs did not explicitly consider the overfitting property of deep neural networks (DNNs). As a result, applying these losses may still suffer from overfitting/memorizing noisy labels as training proceeds. In this paper, we first theoretically analyze the memorization effect and show that a lower-capacity model may perform better on noisy datasets. However, it is non-trivial to design a neural network with the best capacity given an arbitrary task. To circumvent this dilemma, instead of changing the model architecture, we decouple DNNs into an encoder followed by a linear classifier and propose to restrict the function space of a DNN by a representation regularizer. Particularly, we require the distance between two self-supervised features to be positively related to the distance between the corresponding two supervised model outputs. Our proposed framework is easily extendable and can incorporate many other robust loss functions to further improve performance. Extensive experiments and theoretical analyses support our claims. Code is available at github.com/UCSC-REAL/SelfSup_NoisyLabel.