The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labels
Not on the map. Why?
Abstract (arXiv)
Deep neural networks have incredible capacity and expressibility, and can seemingly memorize any training set. This introduces a problem when training in the presence of noisy labels, as the noisy examples cannot be distinguished from clean examples by the end of training. Recent research has dealt with this challenge by utilizing the fact that deep networks seem to memorize clean examples much earlier than noisy examples. Here we report a new empirical result: for each example, when looking at the time it has been memorized by each model in an ensemble of networks, the diversity seen in noisy examples is much larger than the clean examples. We use this observation to develop a new method for noisy labels filtration. The method is based on a statistics of the data, which captures the differences in ensemble learning dynamics between clean and noisy data. We test our method on three tasks: (i) noise amount estimation; (ii) noise filtration; (iii) supervised classification. We show that our method improves over existing baselines in all three tasks using a variety of datasets, noise models, and noise levels. Aside from its improved performance, our method has two other advantages. (i) Simplicity, which implies that no additional hyperparameters are introduced. (ii) Our method is modular: it does not work in an end-to-end fashion, and can therefore be used to clean a dataset for any other future usage.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 2 Oct 2022.
Builds on
43 of 58 references.
References · 43
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
- 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
- 2017A Closer Look at Memorization in Deep Networkscited by 936
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
- 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
- 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
31 more
- 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
- 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
- 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
- 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
- 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
- 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
- 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
- 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
- 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
- 2021Deep learning: a statistical viewpointcited by 239
- 2017Decoupling "when to update" from "how to update"cited by 230
- 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
- 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
- 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
- 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
- 2018Learning to Learn from Noisy Labeled Datacited by 160
- 2021Deep Learning Through the Lens of Example Difficultycited by 139
- 2019Learning From Noisy Labels By Regularized Estimation Of Annotator Confusioncited by 126
- 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
- 2022Selective-Supervised Contrastive Learning with Noisy Labelscited by 112
- 2017Implicit Regularization in Deep Learningcited by 90
- 2020Multi-Objective Interpolation Training for Robustness to Label Noisecited by 86
- 2021Jo-SRC: A Contrastive Approach for Combating Noisy Labelscited by 80
- 2022UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learningcited by 77
- 2021Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labelscited by 69
- 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
- 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 55
- 2018Deep Bilevel Learningcited by 48
- 2018Theory of Curriculum Learning, with Convex Loss Functionscited by 33
- 2019Robust Training with Ensemble Consensuscited by 11
- 2021When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statisticscited by 11
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.