# When does Privileged Information Explain Away Label Noise?

arXiv: https://arxiv.org/abs/2303.01806 · HTML: https://mlsanity.com/paper/2303.01806/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Gradient Starvation: A Learning Proclivity in Neural Networks](https://mlsanity.com/paper/2011.09468/) (2020)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [Learning with Instance-Dependent Label Noise: A Sample Sieve Approach](https://mlsanity.com/paper/2010.02347/) (2020)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Learning From Noisy Large-Scale Datasets With Minimal Supervision](https://mlsanity.com/paper/1701.01619/) (2017)
- [Understanding and Improving Early Stopping for Learning with Noisy Labels](https://mlsanity.com/paper/2106.15853/) (2021)
- [Robust Training under Label Noise by Over-parameterization](https://mlsanity.com/paper/2202.14026/) (2022)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Removing Hidden Confounding by Experimental Grounding](https://mlsanity.com/paper/1810.11646/) (2018)
- [What Do Neural Networks Learn When Trained With Random Labels?](https://mlsanity.com/paper/2006.10455/) (2020)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [Automatic Shortcut Removal for Self-Supervised Representation Learning](https://mlsanity.com/paper/2002.08822/) (2020)
- [Correlated Input-Dependent Label Noise in Large-Scale Image Classification](https://mlsanity.com/paper/2105.10305/) (2021)
- [Deep Learning under Privileged Information Using Heteroscedastic Dropout](https://mlsanity.com/paper/1805.11614/) (2018)
- [High-Dimensional Linear Regression via Implicit Regularization](https://mlsanity.com/paper/1903.09367/) (2019)

## Built on by (most cited first, 10 of 10)

- [URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates](https://mlsanity.com/paper/2307.03810/) (2023)
- [Knockout: A simple way to handle missing inputs](https://mlsanity.com/paper/2405.20448/) (2024)
- [Conformal Prediction with Corrupted Labels: Uncertain Imputation and Robust Re-weighting](https://mlsanity.com/paper/2505.04733/) (2025)
- [Pi-DUAL: Using Privileged Information to Distinguish Clean from Noisy Labels](https://mlsanity.com/paper/2310.06600/) (2023)
- [Rethinking Knowledge Transfer in Learning Using Privileged Information](https://mlsanity.com/paper/2408.14319/) (2024)
- [Uncertainties of Latent Representations in Computer Vision](https://mlsanity.com/paper/2408.14281/) (2024)
- [Noisy Ostracods: A Fine-Grained, Imbalanced Real-World Dataset for Benchmarking Robust Machine Learning and Label Correction Methods](https://mlsanity.com/paper/2412.02313/) (2024)
- [Driving Style Recognition Like an Expert Using Semantic Privileged Information from Large Language Models](https://mlsanity.com/paper/2508.13881/) (2025)
- [When are radiology reports useful for training medical image classifiers?](https://mlsanity.com/paper/2510.24385/) (2025)
- [TextClusterLab: An Integrated Framework for Reliable Text Clustering Studies](https://mlsanity.com/paper/2606.28328/) (2026)

## Abstract (arXiv)

Leveraging privileged information (PI), or features available during training but not at test time, has recently been shown to be an effective method for addressing label noise. However, the reasons for its effectiveness are not well understood. In this study, we investigate the role played by different properties of the PI in explaining away label noise. Through experiments on multiple datasets with real PI (CIFAR-N/H) and a new large-scale benchmark ImageNet-PI, we find that PI is most helpful when it allows networks to easily distinguish clean from noisy data, while enabling a learning shortcut to memorize the noisy examples. Interestingly, when PI becomes too predictive of the target label, PI methods often perform worse than their no-PI baselines. Based on these findings, we propose several enhancements to the state-of-the-art PI methods and demonstrate the potential of PI as a means of tackling label noise. Finally, we show how we can easily combine the resulting PI approaches with existing no-PI techniques designed to deal with label noise.