mlsanityData to 7 Oct 2026

Continuous vs. Discrete Optimization of Deep Neural Networks

arXiv 2107.06608 ↗First posted 14 Jul 2021cs.LG cs.AI cs.NEcited by 34

Not on the map. Why?

Abstract (arXiv)

Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical analysis, but is stylized and disregards computational efficiency. The extent to which it represents gradient descent is an open question in the theory of deep learning. The current paper studies this question. Viewing gradient descent as an approximate numerical solution to the initial value problem of gradient flow, we find that the degree of approximation depends on the curvature around the gradient flow trajectory. We then show that over deep neural networks with homogeneous activations, gradient flow trajectories enjoy favorable curvature, suggesting they are well approximated by gradient descent. This finding allows us to translate an analysis of gradient flow over deep linear neural networks into a guarantee that gradient descent efficiently converges to global minimum almost surely under random initialization. Experiments suggest that over simple deep neural networks, gradient descent with conventional step size is indeed close to gradient flow. We hypothesize that the theory of gradient flows will unravel mysteries behind deep learning.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 51 of its 76 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

51 of 76 references.

References · 51

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  7. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  8. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  9. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  12. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
38 more
  1. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  2. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  3. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  4. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  5. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  6. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  7. 2020Directional convergence and alignment in deep learningcited by 172
  8. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  9. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  10. 2020Implicit Gradient Regularizationcited by 145
  11. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  12. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  13. 2018Stochastic subgradient method converges on tame functionscited by 120
  14. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  15. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  16. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 108
  17. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  18. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  19. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  20. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  21. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  22. 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
  23. 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
  24. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  25. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  26. 2021Implicit Regularization in Tensor Factorizationcited by 43
  27. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  28. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  29. 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
  30. 2018Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscited by 22
  31. 2018Direct Runge-Kutta Discretization Achieves Accelerationcited by 21
  32. 2020Training Linear Neural Networks: Non-Local Convergence and Complexity Resultscited by 19
  33. 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
  34. 2019Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connectionscited by 7
  35. 2017Integration Methods and Accelerated Optimization Algorithmscited by 6
  36. 2018ADMM and Accelerated ADMM as Continuous Dynamical Systemscited by 6
  37. 2019Shadowing Properties of Optimization Algorithmscited by 6
  38. 2021A Convergence Theory Towards Practical Over-parameterized Deep Neural Networkscited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 6; 2024: 6; 2025: 8; 2026: 3.

34 papers.

Sort:

By area:

  1. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  2. 2022Perturbation Analysis of Neural Collapsecited by 34
  3. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  4. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
  5. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  6. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  7. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  8. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  9. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  10. 2023Neural (Tangent Kernel) Collapsecited by 18
  11. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  12. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
22 more
  1. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  2. 2022One-vs-the-Rest Loss to Focus on Important Samples in Adversarial Trainingcited by 7
  3. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  4. 2024Towards Optimal Learning of Language Modelscited by 7
  5. 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
  6. 2026Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Datacited by 5
  7. 2023Meta-Learning Strategies through Value Maximization in Neural Networkscited by 4
  8. 2025A Control Perspective on Training PINNscited by 3
  9. 2025Additive Models Explained: A Computational Complexity Approachcited by 3
  10. 2023A Novel Convolutional Neural Network Architecture with a Continuous Symmetrycited by 2
  11. 2024Continuous-Time Analysis of Adaptive Optimization and Normalizationcited by 2
  12. 2025Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descentcited by 2
  13. 2022Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networkscited by 1
  14. 2024The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rankcited by 1
  15. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  16. 2025Nonlinear dynamics of localization in neural receptive fieldscited by 1
  17. 2025HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuningcited by 1
  18. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
  19. 2024The EarlyBird Gets the WORM: Heuristically Accelerating EarlyBird Convergencecited by 0
  20. 2025Gradient Flow Convergence Guarantee for General Neural Network Architecturescited by 0
  21. 2026Sufficient Conditions for Stability of Minimum-Norm Interpolating Deep ReLU Networkscited by 0
  22. 2026On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspectivecited by 0