mlsanityData to 13 Oct 2026

On a continuous time model of gradient descent dynamics and instability in deep learning

arXiv 2302.01952 ↗First posted 3 Feb 2023stat.ML cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

The recipe behind the success of deep learning has been the combination of neural networks and gradient-based optimization. Understanding the behavior of gradient descent however, and particularly its instability, has lagged behind its empirical success. To add to the theoretical tools available to study gradient descent we propose the principal flow (PF), a continuous time flow that approximates gradient descent dynamics. To our knowledge, the PF is the only continuous flow that captures the divergent and oscillatory behaviors of gradient descent, including escaping local minima and saddle points. Through its dependence on the eigendecomposition of the Hessian the PF sheds light on the recently observed edge of stability phenomena in deep learning. Using our new understanding of instability we propose a learning rate adaptation method which enables us to control the trade-off between training stability and test set evaluation performance.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 63 of its 95 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

63 of 95 references.

References · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,411
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,344
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,129
  9. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 787
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 786
  12. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 755
38 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 737
  2. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 573
  3. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 419
  5. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  6. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 288
  7. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 285
  8. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 282
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  11. 2016Deep Learning without Poor Local Minimacited by 234
  12. 2018Gradient Descent Happens in a Tiny Subspacecited by 213
  13. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 212
  14. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  15. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 175
  16. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  17. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  18. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 153
  19. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  20. 2020Implicit Gradient Regularizationcited by 146
  21. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  22. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  23. 2018The Mechanics of n-Player Differentiable Gamescited by 118
  24. 2017The Numerics of GANscited by 104
  25. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 87
  26. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  27. 2022Understanding the unstable convergence of gradient descentcited by 85
  28. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  29. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
  30. 2017Gradient descent GAN optimization is locally stablecited by 65
  31. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  32. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  33. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  34. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 48
  35. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  36. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  37. 2021Strength of Minibatch Noise in SGDcited by 39
  38. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 3; 2026: 3.

11 papers.

  1. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  2. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  3. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 8
  4. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  5. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  6. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  7. 2023Implicit regularisation in stochastic gradient descent: from single-objective to two-player gamescited by 2
  8. 2025Learning by solving differential equationscited by 1
  9. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
  10. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0
  11. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0