mlsanityData to 14 Oct 2026

Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks

arXiv 2606.06764 ↗First posted 4 Jun 2026stat.ML cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent progress has been made in understanding the statistical generalization performance of gradient descent methods for overparameterized neural networks within the neural tangent kernel (NTK) regime. However, most of the existing work on regression problems is limited to shallow network architectures, leaving a notable gap in the theory of deep neural networks. This paper addresses this gap by presenting a comprehensive generalization analysis for deep ReLU networks trained using gradient descent (GD) and stochastic gradient descent (SGD). Specifically, we establish the first known minimax-optimal rates of excess population risk for both GD and SGD with deep ReLU networks, under the assumption that the network width scales polynomially with respect to the network depth and training sample size. Our results demonstrate that with sufficient width, gradient descent methods for deep ReLU networks can achieve optimal generalization rates on par with kernel methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 59 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

38 of 59 references.

References · 38

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  9. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  10. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  11. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  12. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
26 more
  1. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  2. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  3. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  4. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  5. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  6. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  7. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  8. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  9. 2020Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Networkcited by 38
  10. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  11. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  12. 2018Learning with SGD and Random Featurescited by 33
  13. 2023Generalization Ability of Wide Neural Networks on ℝcited by 30
  14. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  15. 2019Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problemscited by 24
  16. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  17. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  18. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  19. 2023Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networkscited by 9
  20. 2016Optimal Rates for Multi-pass Stochastic Gradient Methodscited by 8
  21. 2024Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methodscited by 7
  22. 2022Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networkscited by 5
  23. 2024Generalization Analysis of Online Stochastic Gradient Descent for Overparameterized Two-Layer Neural Networkscited by 4
  24. 2025Random feature approximation for general spectral methodscited by 2
  25. 2026Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networkscited by 1
  26. 2026Optimization Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noisecited by 1

Built on by

Not cited yet.