mlsanityData to 7 Oct 2026

Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty

arXiv 2209.09658 ↗First posted 19 Sep 2022cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network can be well approximated by its linearization around initialization. Here we investigate the comparative effect of the lazy (linear) and feature learning (non-linear) regimes on subgroups of examples based on their difficulty. Specifically, we show that easier examples are given more weight in feature learning mode, resulting in faster training compared to more difficult ones. In other words, the non-linear dynamics tends to sequentialize the learning of examples of increasing difficulty. We illustrate this phenomenon across different ways to quantify example difficulty, including c-score, label noise, and in the presence of easy-to-learn spurious correlations. Our results reveal a new understanding of how deep networks prioritize resources across example difficulty.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 41 of its 52 references that are in the map. First posted on arXiv 19 Sep 2022.

Builds on

41 of 52 references.

References · 41

  1. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2018On the Spectral Bias of Neural Networkscited by 1,321
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2018On Lazy Training in Differentiable Programmingcited by 671
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  11. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  12. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
29 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  2. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  8. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2021Deep learning: a statistical viewpointcited by 239
  11. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  12. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  13. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  14. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  15. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  16. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  17. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  18. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  19. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  20. 2019How do infinite width bounded norm networks look in function space?cited by 105
  21. 2020When Do Curricula Work?cited by 105
  22. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  23. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  24. 2021What can linearized neural networks actually say about generalization?cited by 49
  25. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  26. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  27. 2020Temperature check: theory and practice for training models with softmax-cross-entropy lossescited by 31
  28. 2020Geometric compression of invariant manifolds in neural netscited by 28
  29. 2020Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNscited by 10

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 1.

5 papers.

  1. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  2. 2024Are Sparse Neural Networks Better Hard Sample Learners?cited by 1
  3. 2024Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmarkcited by 1
  4. 2024Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networkscited by 0
  5. 2025Flexible inference for animal learning rules using neural networkscited by 0