mlsanityData to 7 Oct 2026

Deep Learning Through the Lens of Example Difficulty

arXiv 2106.09647 ↗First posted 17 Jun 2021cs.LG stat.MLcited by 139

Not on the map. Why?

Abstract (arXiv)

Existing work on understanding deep learning often employs measures that compress all data-dependent information into a few numbers. In this work, we adopt a perspective based on the role of individual examples. We introduce a measure of the computational difficulty of making a prediction for a given input: the (effective) prediction depth. Our extensive investigation reveals surprising yet simple relationships between the prediction depth of a given input and the model's uncertainty, confidence, accuracy and speed of learning for that data point. We further categorize difficult examples into three interpretable groups, demonstrate how these groups are processed differently inside deep models and showcase how this understanding allows us to improve prediction accuracy. Insights from our study lead to a coherent view of a number of separately reported phenomena in the literature: early layers generalize while later layers memorize; early layers converge faster and networks learn easy data and simple functions first.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

53 of 69 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2017A Closer Look at Memorization in Deep Networkscited by 936
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
38 more
  1. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2018Universal Transformerscited by 613
  4. 2017Exploring Generalization in Deep Learningcited by 595
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  8. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  9. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  11. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  12. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  13. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  14. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  15. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  16. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  17. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  18. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  19. 2018Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learningcited by 203
  20. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  21. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  22. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  23. 2020Characterising Bias in Compressed Modelscited by 162
  24. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  25. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  26. 2017Generalization in Deep Learningcited by 146
  27. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  28. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  29. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
  30. 2020Predicting Neural Network Accuracy from Weightscited by 107
  31. 2020When Do Curricula Work?cited by 105
  32. 2020Estimating Example Difficulty Using Variance of Gradientscited by 96
  33. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  34. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  35. 2019Generalization bounds for deep convolutional neural networkscited by 58
  36. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  37. 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 55
  38. 2020Deep k-NN for Noisy Labelscited by 55

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 31; 2023: 35; 2024: 33; 2025: 20; 2026: 11.

139 papers.

Sort:

By area:

  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  3. 2023On Memorization in Diffusion Modelscited by 122
  4. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  5. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  6. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  7. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  8. 2023Can Neural Network Memorization Be Localized?cited by 66
  9. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  10. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  11. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  12. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Modelscited by 36
38 more
  1. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  2. 2023Data Valuation Without Training of a Modelcited by 35
  3. 2022The Effect of Task Ordering in Continual Learningcited by 30
  4. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  5. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  6. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 25
  7. 2021Intriguing Properties of Input-dependent Randomized Smoothingcited by 24
  8. 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
  9. 2023Blockwise Self-Supervised Learning at Scalecited by 22
  10. 2023Task-Aware Information Routing from Common Representation Space in Lifelong Learningcited by 22
  11. 2022Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networkscited by 21
  12. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  13. 2021The Impact of Reinitialization on Generalization in Convolutional Neural Networkscited by 20
  14. 2022Nearest Class-Center Simplification through Intermediate Layerscited by 20
  15. 2022Selective Prediction via Training Dynamicscited by 20
  16. 2023On the Variance of Neural Network Training with respect to Test Sets and Distributionscited by 20
  17. 2025Uncovering Memorization Effect in the Presence of Spurious Correlationscited by 20
  18. 2021A Tale Of Two Long Tailscited by 19
  19. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  20. 2022py-irt: A Scalable Item Response Theory Library for Pythoncited by 19
  21. 2023How to prepare your task head for finetuningcited by 18
  22. 2023Late Stopping: Avoiding Confidently Learning from Mislabeled Examplescited by 17
  23. 2023Backdoor Federated Learning by Poisoning Backdoor-Critical Layerscited by 16
  24. 2024Energy-based Automated Model Evaluationcited by 16
  25. 2022Layer Ensembles: A Single-Pass Uncertainty Estimation in Deep Learning for Segmentationcited by 15
  26. 2023Learning Sample Difficulty from Pre-trained Models for Reliable Predictioncited by 15
  27. 2024Cascade-Aware Training of Language Modelscited by 15
  28. 2022When Does Re-initialization Work?cited by 12
  29. 2022CrossSplit: Mitigating Label Noise Memorization through Data Splittingcited by 12
  30. 2023FedDefender: Client-Side Attack-Tolerant Federated Learningcited by 12
  31. 2024Dataset Difficulty and the Role of Inductive Biascited by 12
  32. 2024Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlationscited by 12
  33. 2025From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularizationcited by 12
  34. 2021On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Trainingcited by 11
  35. 2023Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamicscited by 11
  36. 2023A Bag-of-Prototypes Representation for Dataset-Level Applicationscited by 11
  37. 2023Bias in Pruned Vision Models: In-Depth Analysis and Countermeasurescited by 11
  38. 2024Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AIcited by 11

50 most cited of 139 All 139 (JSON)