mlsanityData to 14 Oct 2026

Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis

arXiv 2210.15898 ↗First posted 28 Oct 2022cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We derive and solve an ``Equation of Motion'' (EoM) for deep neural networks (DNNs), a differential equation that precisely describes the discrete learning dynamics of DNNs. Differential equations are continuous but have played a prominent role even in the study of discrete optimization (gradient descent (GD) algorithms). However, there still exist gaps between differential equations and the actual learning dynamics of DNNs due to discretization error. In this paper, we start from gradient flow (GF) and derive a counter term that cancels the discretization error between GF and GD. As a result, we obtain EoM, a continuous differential equation that precisely describes the discrete learning dynamics of GD. We also derive discretization error to show to what extent EoM is precise. In addition, we apply EoM to two specific cases: scale- and translation-invariant layers. EoM highlights differences between continuous-time and discrete-time GD, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD. Our experimental results support our theoretical findings.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 32 of its 73 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

32 of 73 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  7. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  8. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  9. 2017Stein Variational Gradient Descent as Gradient Flowcited by 152
  10. 2020Implicit Gradient Regularizationcited by 146
  11. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  12. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
20 more
  1. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  2. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  3. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  4. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  5. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  6. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  7. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  8. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
  9. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  10. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  11. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  12. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  13. 2019Continuous Time Analysis of Momentum Methodscited by 32
  14. 2019Online Normalization for Training Neural Networkscited by 29
  15. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  16. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  17. 2019Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximationcited by 15
  18. 2020Spherical Perspective on Learning with Normalization Layerscited by 10
  19. 2018Optimal convergence rates for Nesterov accelerationcited by 8
  20. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 3; 2026: 4.

13 papers.

Sort:

By area:

  1. 2023On the Implicit Bias of Adamcited by 27
  2. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  3. 2025How Memory in Optimization Algorithms Implicitly Modifies the Losscited by 6
  4. 2023Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descentcited by 5
  5. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  6. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  7. 2026Optimal Learning Rate Schedule for Balancing Effort and Performancecited by 1
  8. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
  9. 2024Error dynamics of mini-batch gradient descent with random reshuffling for least squares regressioncited by 0
  10. 2025Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysiscited by 0
  11. 2025A Control-Barrier-Function-Based Algorithm for Policy Adaptation in Reinforcement Learningcited by 0
  12. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0
1 more
  1. 2026A Journey to the Edge of Stabilitycited by 0