# When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statistics

arXiv: https://arxiv.org/abs/2105.08997 · HTML: https://mlsanity.com/paper/2105.08997/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 58 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [What Do Neural Networks Learn When Trained With Random Labels?](https://mlsanity.com/paper/2006.10455/) (2020)
- [Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets](https://mlsanity.com/paper/1905.10854/) (2019)
- [How hard can it be? Estimating the difficulty of visual search in an image](https://mlsanity.com/paper/1705.08280/) (2017)
- [Hold me tight! Influence of discriminative features on deep network boundaries](https://mlsanity.com/paper/2002.06349/) (2020)
- [Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networks](https://mlsanity.com/paper/2105.05553/) (2021)

## Built on by (most cited first, 11 of 11)

- [Continual Learning: Applications and the Road Forward](https://mlsanity.com/paper/2311.11908/) (2023)
- [The Grammar-Learning Trajectories of Neural Language Models](https://mlsanity.com/paper/2109.06096/) (2021)
- [Demystifying Disagreement-on-the-Line in High Dimensions](https://mlsanity.com/paper/2301.13371/) (2023)
- [Task Discovery: Finding the Tasks that Neural Networks Generalize on](https://mlsanity.com/paper/2212.00261/) (2022)
- [A Lightweight Measure of Classification Difficulty from Application Dataset Characteristics](https://mlsanity.com/paper/2404.05981/) (2024)
- [HODA: Hardness-Oriented Detection of Model Extraction Attacks](https://mlsanity.com/paper/2106.11424/) (2021)
- [The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labels](https://mlsanity.com/paper/2210.00583/) (2022)
- [United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfit](https://mlsanity.com/paper/2310.11077/) (2023)
- [Robustness as an Emergent Property of Task Performance](https://mlsanity.com/paper/2602.03344/) (2026)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)
- [Efficient Neural Network Model Selection for Few-Class Application Datasets](https://mlsanity.com/paper/2606.19712/) (2026)

## Abstract (arXiv)

Although a plethora of architectural variants for deep classification has been introduced over time, recent works have found empirical evidence towards similarities in their training process. It has been hypothesized that neural networks converge not only to similar representations, but also exhibit a notion of empirical agreement on which data instances are learned first. Following in the latter works' footsteps, we define a metric to quantify the relationship between such classification agreement over time, and posit that the agreement phenomenon can be mapped to core statistics of the investigated dataset. We empirically corroborate this hypothesis across the CIFAR10, Pascal, ImageNet and KTH-TIPS2 datasets. Our findings indicate that agreement seems to be independent of specific architectures, training hyper-parameters or labels, albeit follows an ordering according to image statistics.