mlsanityData to 14 Oct 2026

Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets

arXiv 1905.10854 ↗First posted 26 May 2019cs.LG stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries -- models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Thus, when fixing the architecture, we show synthetic datasets where this pattern ceases to exist. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 14; 2023: 12; 2024: 7; 2025: 3; 2026: 9.

56 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  3. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 133
  4. 2022Fusing finetuned models for better pretrainingcited by 110
  5. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 101
  6. 2023Continual Learning: Applications and the Road Forwardcited by 59
  7. 2022Evaluation of post-hoc interpretability methods in time-series classificationcited by 53
  8. 2021Do Input Gradients Highlight Discriminative Features?cited by 52
  9. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  10. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  11. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  12. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
38 more
  1. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  2. 2023The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifoldcited by 21
  3. 2021Memorization and Generalization in Neural Code Intelligence Modelscited by 19
  4. 2024Label-Efficient Model Selection for Text Generationcited by 19
  5. 2024Objective drives the consistency of representational similarity across datasetscited by 18
  6. 2023How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budgetcited by 17
  7. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
  8. 2024Memorization in deep learning: A surveycited by 15
  9. 2021Mediators in Determining what Processing BERT Performs Firstcited by 14
  10. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  11. 2024Dataset Difficulty and the Role of Inductive Biascited by 12
  12. 2021When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statisticscited by 11
  13. 2022LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classificationcited by 11
  14. 2022Simplicity Bias Leads to Amplified Performance Disparitiescited by 11
  15. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  16. 2023Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learningcited by 10
  17. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  18. 2022Learning from Training Dynamics: Identifying Mislabeled Data Beyond Manually Designed Featurescited by 9
  19. 2022Deep Double Descent via Smooth Interpolationcited by 8
  20. 2024Predicting the Susceptibility of Examples to Catastrophic Forgettingcited by 7
  21. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  22. 2023Influence Scores at Scale for Efficient Language Data Samplingcited by 5
  23. 2023Unraveling the Key Components of OOD Generalization via Diversificationcited by 4
  24. 2024Unforgettable Generalization in Language Modelscited by 4
  25. 2021HODA: Hardness-Oriented Detection of Model Extraction Attackscited by 3
  26. 2022A Tour of Visualization Techniques for Computer Vision Datasetscited by 3
  27. 2022The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labelscited by 3
  28. 2025Causes and Consequences of Representational Similarity in Machine Learning Modelscited by 3
  29. 2023Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applicationscited by 2
  30. 2023Data-Centric Diet: Effective Multi-center Dataset Pruning for Medical Image Segmentationcited by 2
  31. 2023United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfitcited by 2
  32. 2025AttackPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agentscited by 2
  33. 2026Robustness as an Emergent Property of Task Performancecited by 2
  34. 2021Statistical Measures For Defining Curriculum Scoring Functioncited by 1
  35. 2022The Lean Data Scientist: Recent Advances towards Overcoming the Data Bottleneckcited by 1
  36. 2025An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Modelscited by 1
  37. 2021When Neural Networks Using Different Sensors Create Similar Featurescited by 0
  38. 2023Do Smaller Language Models Answer Contextualised Questions Through Memorisation Or Generalisation?cited by 0

50 most cited of 56 All 56 (JSON)