# Correlated Input-Dependent Label Noise in Large-Scale Image Classification

arXiv: https://arxiv.org/abs/2105.10305 · HTML: https://mlsanity.com/paper/2105.10305/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images](https://mlsanity.com/paper/1808.01097/) (2018)

## Built on by (most cited first, 30 of 37)

- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Uncertainty Estimation by Fisher Information-based Evidential Deep Learning](https://mlsanity.com/paper/2303.02045/) (2023)
- [Learning with Neighbor Consistency for Noisy Labels](https://mlsanity.com/paper/2202.02200/) (2022)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning](https://mlsanity.com/paper/2202.05613/) (2022)
- [Improving Image Recognition by Retrieving from Web-Scale Image-Text Data](https://mlsanity.com/paper/2304.05173/) (2023)
- [Transfer and Marginalize: Explaining Away Label Noise with Privileged Information](https://mlsanity.com/paper/2202.09244/) (2022)
- [Constrained Instance and Class Reweighting for Robust Learning under Label Noise](https://mlsanity.com/paper/2111.05428/) (2021)
- [Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts](https://mlsanity.com/paper/2302.06495/) (2023)
- [Massively Scaling Heteroscedastic Classifiers](https://mlsanity.com/paper/2301.12860/) (2023)
- [When does Privileged Information Explain Away Label Noise?](https://mlsanity.com/paper/2303.01806/) (2023)
- [Deep Classifiers with Label Noise Modeling and Distance Awareness](https://mlsanity.com/paper/2110.02609/) (2021)
- [Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shifts](https://mlsanity.com/paper/2403.05600/) (2024)
- [Selectively Answering Visual Questions](https://mlsanity.com/paper/2406.00980/) (2024)
- [Knockout: A simple way to handle missing inputs](https://mlsanity.com/paper/2405.20448/) (2024)
- [Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification](https://mlsanity.com/paper/2508.04457/) (2025)
- [Cold Posteriors through PAC-Bayes](https://mlsanity.com/paper/2206.11173/) (2022)
- [Logistic-Normal Likelihoods for Heteroscedastic Label Noise](https://mlsanity.com/paper/2304.02849/) (2023)
- [Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality](https://mlsanity.com/paper/2305.12811/) (2023)
- [Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classification](https://mlsanity.com/paper/2504.13111/) (2025)
- [Tackling Instance-Dependent Label Noise with Dynamic Distribution Calibration](https://mlsanity.com/paper/2210.05126/) (2022)
- [Pushing the Accuracy-Group Robustness Frontier with Introspective Self-play](https://mlsanity.com/paper/2302.05807/) (2023)
- [Pi-DUAL: Using Privileged Information to Distinguish Clean from Noisy Labels](https://mlsanity.com/paper/2310.06600/) (2023)
- [Transferable Candidate Proposal with Bounded Uncertainty](https://mlsanity.com/paper/2312.04604/) (2023)
- [Rethinking Approximate Gaussian Inference in Classification](https://mlsanity.com/paper/2502.03366/) (2025)

## Abstract (arXiv)

Large scale image classification datasets often contain noisy labels. We take a principled probabilistic approach to modelling input-dependent, also known as heteroscedastic, label noise in these datasets. We place a multivariate Normal distributed latent variable on the final hidden layer of a neural network classifier. The covariance matrix of this latent variable, models the aleatoric uncertainty due to label noise. We demonstrate that the learned covariance structure captures known sources of label noise between semantically similar and co-occurring classes. Compared to standard neural network training and other baselines, we show significantly improved accuracy on Imagenet ILSVRC 2012 79.3% (+2.6%), Imagenet-21k 47.0% (+1.1%) and JFT 64.7% (+1.6%). We set a new state-of-the-art result on WebVision 1.0 with 76.6% top-1 accuracy. These datasets range from over 1M to over 300M training examples and from 1k classes to more than 21k classes. Our method is simple to use, and we provide an implementation that is a drop-in replacement for the final fully-connected layer in a deep classifier.