mlsanityData to 7 Oct 2026

One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks

arXiv 2105.00262 ↗First posted 1 May 2021stat.ML cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

There has been a recent surge of interest in understanding the convergence of gradient descent (GD) and stochastic gradient descent (SGD) in overparameterized neural networks. Most previous works assume that the training data is provided a priori in a batch, while less attention has been paid to the important setting where the training data arrives in a stream. In this paper, we study the streaming data setup and show that with overparamterization and random initialization, the prediction error of two-layer neural networks under one-pass SGD converges in expectation. The convergence rate depends on the eigen-decomposition of the integral operator associated with the so-called neural tangent kernel (NTK). A key step of our analysis is to show a random kernel function converges to the NTK with high probability using the VC dimension and McDiarmid's inequality.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 1 May 2021.

Builds on

19 of 40 references.

References · 19

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  10. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  11. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  12. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
7 more
  1. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  2. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  3. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  4. 2018How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?cited by 19
  5. 2019Can SGD Learn Recurrent Neural Networks with Provable Generalization?cited by 17
  6. 2019The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networkscited by 6
  7. 2020A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

3 papers.

  1. 2022On the Global Convergence of Fitted Q-Iteration with Two-layer Neural Network Parametrizationcited by 3
  2. 2022Mutual Information Learned Regressor: an Information-theoretic Viewpoint of Training Regression Systemscited by 0
  3. 2026Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Modelscited by 0