mlsanityData to 7 Oct 2026

Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime

arXiv 2006.12297 ↗First posted 22 Jun 2020stat.ML cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

We analyze the convergence of the averaged stochastic gradient descent for overparameterized two-layer neural networks for regression problems. It was recently found that a neural tangent kernel (NTK) plays an important role in showing the global convergence of gradient-based methods under the NTK regime, where the learning dynamics for overparameterized neural networks can be almost characterized by that for the associated reproducing kernel Hilbert space (RKHS). However, there is still room for a convergence rate analysis in the NTK regime. In this study, we show that the averaged stochastic gradient descent can achieve the minimax optimal convergence rate, with the global convergence guarantee, by exploiting the complexities of the target function and the RKHS associated with the NTK. Moreover, we show that the target function specified by the NTK of a ReLU network can be learned at the optimal convergence rate through a smooth approximation of a ReLU network under certain conditions.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 3; 2023: 4; 2024: 8; 2025: 5; 2026: 5.

35 papers.

Sort:

By area:

  1. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  2. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  3. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  4. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  5. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  6. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  7. 2021A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?cited by 14
  8. 2021Learning curves for Gaussian process regression with power-law priors and targetscited by 14
  9. 2021Understanding Square Loss in Training Overparametrized Neural Network Classifierscited by 14
  10. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  11. 2022Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descentcited by 11
  12. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
23 more
  1. 2022Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditionscited by 9
  2. 2021Deep Networks Provably Classify Data on Curvescited by 6
  3. 2024How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learningcited by 6
  4. 2021On the Double Descent of Random Features Models Trained with SGDcited by 5
  5. 2025A Gaussian Process View on Observation Noise and Initialization in Wide Neural Networkscited by 5
  6. 2021Universal scaling laws in the gradient descent training of neural networkscited by 4
  7. 2023Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularizationcited by 4
  8. 2024Generalization Analysis of Online Stochastic Gradient Descent for Overparameterized Two-Layer Neural Networkscited by 4
  9. 2025Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classificationcited by 3
  10. 2023On the rate of convergence of an over-parametrized Transformer classifier learned by gradient descentcited by 2
  11. 2024Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regressioncited by 2
  12. 2025Random feature approximation for general spectral methodscited by 2
  13. 2026Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networkscited by 1
  14. 2021Painless step size adaptation for SGDcited by 0
  15. 2021Neural Optimization Kernel: Towards Robust Deep Learningcited by 0
  16. 2023Random feature approximation for general spectral methodscited by 0
  17. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  18. 2025Statistically guided deep learningcited by 0
  19. 2025Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Conditioncited by 0
  20. 2026On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inferencecited by 0
  21. 2026Random Features for Operator-Valued Kernels: Bridging Kernel Methods and Neural Operatorscited by 0
  22. 2026Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networkscited by 0
  23. 2026Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descentcited by 0