mlsanityData to 7 Oct 2026

The Dynamics of Gradient Descent for Overparametrized Neural Networks

arXiv 2105.06569 ↗First posted 13 May 2021cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We consider the dynamics of gradient descent (GD) in overparameterized single hidden layer neural networks with a squared loss function. Recently, it has been shown that, under some conditions, the parameter values obtained using GD achieve zero training error and generalize well if the initial conditions are chosen appropriately. Here, through a Lyapunov analysis, we show that the dynamics of neural network weights under GD converge to a point which is close to the minimum norm solution subject to the condition that there is no training error when using the linear approximation to the neural network. To illustrate the application of this result, we show that the GD converges to a prediction function that generalizes well, thereby providing an alternative proof of the generalization results in Arora et al. (2019).

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 13 May 2021.

Builds on

18 of 26 references.

References · 18

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  8. 2019Benign Overfitting in Linear Regressioncited by 544
  9. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  10. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2019Linearized two-layers neural networks in high dimensioncited by 155
6 more
  1. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  2. 2017Implicit Regularization in Deep Learningcited by 90
  3. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  4. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  5. 2018Risk and parameter convergence of logistic regressioncited by 81
  6. 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 3; 2025: 1.

11 papers.

  1. 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
  2. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  3. 2023Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequenciescited by 7
  4. 2022On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Modelcited by 6
  5. 2022Reinforcement Learning with Unbiased Policy Evaluation and Linear Function Approximationcited by 4
  6. 2023Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learningcited by 4
  7. 2023Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimescited by 4
  8. 2024Performance of NPG in Countable State-Space Average-Cost RLcited by 4
  9. 2023Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networkscited by 2
  10. 2024Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?cited by 0
  11. 2024Convergence Guarantees for the DeepWalk Embedding on Block Modelscited by 0