mlsanityData to 7 Oct 2026

Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width

arXiv 2302.12250 ↗First posted 23 Feb 2023cs.LG cond-mat.dis-nncited by 19

Not on the map. Why?

Abstract (arXiv)

We systematically analyze optimization dynamics in deep neural networks (DNNs) trained with stochastic gradient descent (SGD) and study the effect of learning rate η, depth d, and width w of the neural network. By analyzing the maximum eigenvalue λHt of the Hessian of the loss, which is a measure of sharpness of the loss landscape, we find that the dynamics can show four distinct regimes: (i) an early time transient regime, (ii) an intermediate saturation regime, (iii) a progressive sharpening regime, and (iv) a late time ``edge of stability" regime. The early and intermediate regimes (i) and (ii) exhibit a rich phase diagram depending on η equiv c / λ0H , d, and w. We identify several critical values of c, which separate qualitatively distinct phenomena in the early time dynamics of training loss and sharpness. Notably, we discover the opening up of a ``sharpness reduction" phase, where sharpness decreases at early times, as d and 1/w are increased.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

51 of 71 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  11. 2017Deep Neural Networks as Gaussian Processescited by 617
  12. 2017Self-Normalizing Neural Networkscited by 591
38 more
  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  5. 2017Three Factors Influencing Minima in SGDcited by 270
  6. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  7. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  8. 2014Qualitatively characterizing neural network optimization problemscited by 186
  9. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  10. 2021The Principles of Deep Learning Theorycited by 169
  11. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  12. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  13. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  14. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  15. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  16. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  17. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  18. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  19. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  20. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  21. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  22. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  23. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  24. 2020Neural Kernels Without Tangentscited by 56
  25. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  26. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  27. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  28. 2020The Two Regimes of Deep Network Trainingcited by 30
  29. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  30. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  31. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  32. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  33. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  34. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  35. 2018On the Selection of Initialization and Activation Function for Deep Neural Networkscited by 11
  36. 2020Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Trainingcited by 11
  37. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  38. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 4; 2026: 5.

19 papers.

Sort:

  1. 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 45
  2. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  3. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  4. 2024Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transitioncited by 16
  5. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  6. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  7. 2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
  8. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 7
  9. 2024Understanding Token Probability Encoding in Output Embeddingscited by 6
  10. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  11. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  12. 2024HyperbolicLR: Epoch insensitive learning rate schedulercited by 3
7 more
  1. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  2. 2025Topological Invariance and Breakdown in Learningcited by 1
  3. 2026Noise-Driven Exploration and Transient Freezing Select Flat Minima in Stochastic Gradient Descentcited by 1
  4. 2025Enlightenment Period Improving DNN Performancecited by 0
  5. 2025An Equivariance Toolbox for Learning Dynamicscited by 0
  6. 2026How the Hessian-Spectrum of Neural Networks Depends on Datacited by 0
  7. 2026A Journey to the Edge of Stabilitycited by 0