# Learning from Noisy Labels with Distillation

arXiv: https://arxiv.org/abs/1703.02391 · HTML: https://mlsanity.com/paper/1703.02391/

## Key facts

- Cited by 161 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2017.

## Built on by (most cited first, 30 of 161)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2101.10979/) (2021)
- [Towards Understanding Knowledge Distillation](https://mlsanity.com/paper/2105.13093/) (2021)
- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [Learning with Feature-Dependent Label Noise: A Progressive Approach](https://mlsanity.com/paper/2103.07756/) (2021)
- [Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers](https://mlsanity.com/paper/2103.16553/) (2021)
- [Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training](https://mlsanity.com/paper/2301.02280/) (2023)
- [SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2108.06227/) (2021)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Input Similarity from the Neural Network Perspective](https://mlsanity.com/paper/2102.05262/) (2021)
- [Learning Fast Sample Re-weighting Without Reward Data](https://mlsanity.com/paper/2109.03216/) (2021)
- [The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation](https://mlsanity.com/paper/2206.03014/) (2022)
- [Robust Cross-Modal Representation Learning with Progressive Self-Distillation](https://mlsanity.com/paper/2204.04588/) (2022)
- [ProMix: Combating Label Noise via Maximizing Clean Sample Utility](https://mlsanity.com/paper/2207.10276/) (2022)
- [Robust Contrastive Learning against Noisy Views](https://mlsanity.com/paper/2201.04309/) (2022)
- [Understanding the role of importance weighting for deep learning](https://mlsanity.com/paper/2103.15209/) (2021)
- [Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning](https://mlsanity.com/paper/2502.01968/) (2025)
- [Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation](https://mlsanity.com/paper/2112.09445/) (2021)
- [Joint Negative and Positive Learning for Noisy Labels](https://mlsanity.com/paper/2104.06574/) (2021)
- [Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalization](https://mlsanity.com/paper/2208.11351/) (2022)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [Estimating Instance-dependent Bayes-label Transition Matrix using a Deep Neural Network](https://mlsanity.com/paper/2105.13001/) (2021)
- [Continual Learning on Noisy Data Streams via Self-Purified Replay](https://mlsanity.com/paper/2110.07735/) (2021)
- [BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency](https://mlsanity.com/paper/2303.12419/) (2023)
- [Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models](https://mlsanity.com/paper/2311.11202/) (2023)
- [Sum of Ranked Range Loss for Supervised Learning](https://mlsanity.com/paper/2106.03300/) (2021)
- [Few-shot Learning with Noisy Labels](https://mlsanity.com/paper/2204.05494/) (2022)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)

## Abstract (arXiv)

The ability of learning from noisy labels is very useful in many visual recognition tasks, as a vast amount of data with noisy labels are relatively easy to obtain. Traditionally, the label noises have been treated as statistical outliers, and approaches such as importance re-weighting and bootstrap have been proposed to alleviate the problem. According to our observation, the real-world noisy labels exhibit multi-mode characteristics as the true labels, rather than behaving like independent random outliers. In this work, we propose a unified distillation framework to use side information, including a small clean dataset and label relations in knowledge graph, to "hedge the risk" of learning from noisy labels. Furthermore, unlike the traditional approaches evaluated based on simulated label noises, we propose a suite of new benchmark datasets, in Sports, Species and Artifacts domains, to evaluate the task of learning from noisy labels in the practical setting. The empirical study demonstrates the effectiveness of our proposed method in all the domains.