mlsanityData to 7 Oct 2026

When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statistics

arXiv 2105.08997 ↗First posted 19 May 2021cs.LG cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Although a plethora of architectural variants for deep classification has been introduced over time, recent works have found empirical evidence towards similarities in their training process. It has been hypothesized that neural networks converge not only to similar representations, but also exhibit a notion of empirical agreement on which data instances are learned first. Following in the latter works' footsteps, we define a metric to quantify the relationship between such classification agreement over time, and posit that the agreement phenomenon can be mapped to core statistics of the investigated dataset. We empirically corroborate this hypothesis across the CIFAR10, Pascal, ImageNet and KTH-TIPS2 datasets. Our findings indicate that agreement seems to be independent of specific architectures, training hyper-parameters or labels, albeit follows an ordering according to image statistics.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 23 of its 58 references that are in the map. First posted on arXiv 19 May 2021.

Builds on

23 of 58 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  9. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  10. 2017A Closer Look at Memorization in Deep Networkscited by 936
  11. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  12. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
11 more
  1. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  2. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  3. 2019Human uncertainty makes classification more robustcited by 231
  4. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  5. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  6. 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
  7. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 58
  8. 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 55
  9. 2017How hard can it be? Estimating the difficulty of visual search in an imagecited by 52
  10. 2020Hold me tight! Influence of discriminative features on deep network boundariescited by 31
  11. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 3; 2024: 1; 2025: 1; 2026: 2.

11 papers.

  1. 2023Continual Learning: Applications and the Road Forwardcited by 59
  2. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  3. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  4. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  5. 2024A Lightweight Measure of Classification Difficulty from Application Dataset Characteristicscited by 4
  6. 2022The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labelscited by 3
  7. 2021HODA: Hardness-Oriented Detection of Model Extraction Attackscited by 2
  8. 2023United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfitcited by 2
  9. 2026Robustness as an Emergent Property of Task Performancecited by 2
  10. 2025LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Trainingcited by 0
  11. 2026Efficient Neural Network Model Selection for Few-Class Application Datasetscited by 0