# Relation Modeling and Distillation for Learning with Noisy Labels

arXiv: https://arxiv.org/abs/2405.19606 · HTML: https://mlsanity.com/paper/2405.19606/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 26 of 26)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [SELF: Learning to Filter Noisy Labels with Self-Ensembling](https://mlsanity.com/paper/1910.01842/) (2019)
- [NLNL: Negative Learning for Noisy Labels](https://mlsanity.com/paper/1908.07387/) (2019)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Deep Self-Learning From Noisy Labels](https://mlsanity.com/paper/1908.02160/) (2019)
- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [Curriculum Loss: Robust Learning and Generalization against Label Corruption](https://mlsanity.com/paper/1905.10045/) (2019)
- [Error-Bounded Correction of Noisy Labels](https://mlsanity.com/paper/2011.10077/) (2020)
- [Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images](https://mlsanity.com/paper/1905.01273/) (2019)
- [Fine-Grained Classification with Noisy Labels](https://mlsanity.com/paper/2303.02404/) (2023)
- [Bootstrapping the Relationship Between Images and Their Clean and Noisy Labels](https://mlsanity.com/paper/2210.08826/) (2022)
- [Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment](https://mlsanity.com/paper/2211.13264/) (2022)
- [On Mask-based Image Set Desensitization with Recognition Support](https://mlsanity.com/paper/2312.08975/) (2023)

## Abstract (arXiv)

Learning with noisy labels has become an effective strategy for enhancing the robustness of models, which enables models to better tolerate inaccurate data. Existing methods either focus on optimizing the loss function to mitigate the interference from noise, or design procedures to detect potential noise and correct errors. However, their effectiveness is often compromised in representation learning due to the dilemma where models overfit to noisy labels. To address this issue, this paper proposes a relation modeling and distillation framework that models inter-sample relationships via self-supervised learning and employs knowledge distillation to enhance understanding of latent associations, which mitigate the impact of noisy labels. Specifically, the proposed method, termed RMDNet, includes two main modules, where the relation modeling (RM) module implements the contrastive learning technique to learn representations of all data, an unsupervised approach that effectively eliminates the interference of noisy tags on feature extraction. The relation-guided representation learning (RGRL) module utilizes inter-sample relation learned from the RM module to calibrate the representation distribution for noisy samples, which is capable of improving the generalization of the model in the inference phase. Notably, the proposed RMDNet is a plug-and-play framework that can integrate multiple methods to its advantage. Extensive experiments were conducted on two datasets, including performance comparison, ablation study, in-depth analysis and case study. The results show that RMDNet can learn discriminative representations for noisy data, which results in superior performance than the existing methods.