mlsanityData to 14 Oct 2026

Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networks

arXiv 2305.16891 ↗First posted 26 May 2023cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, significant progress has been made in understanding the generalization of neural networks (NNs) trained by gradient descent (GD) using the algorithmic stability approach. However, most of the existing research has focused on one-hidden-layer NNs and has not addressed the impact of different network scaling parameters. In this paper, we greatly extend the previous work citelei2022stability,richards2021stability by conducting a comprehensive stability and generalization analysis of GD for multi-layer NNs. For two-layer NNs, our results are established under general network scaling parameters, relaxing previous conditions. In the case of three-layer NNs, our technical contribution lies in demonstrating its nearly co-coercive property by utilizing a novel induction strategy that thoroughly explores the effects of over-parameterization. As a direct application of our general findings, we derive the excess risk rate of O(1/√n) for GD algorithms in both two-layer and three-layer NNs. This sheds light on sufficient or necessary conditions for under-parameterized and over-parameterized NNs trained by GD to attain the desired risk rate of O(1/√n). Moreover, we demonstrate that as the scaling parameter increases or the network complexity decreases, less over-parameterization is required for GD to achieve the desired error rates. Additionally, under a low-noise condition, we obtain a fast risk rate of O(1/n) for GD in both two-layer and three-layer NNs.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

33 of 43 references.

References · 33

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  11. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  12. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
21 more
  1. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  4. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  5. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  7. 2021Deep learning: a statistical viewpointcited by 240
  8. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  9. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  10. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 119
  11. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  12. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 92
  13. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  14. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  15. 2019Generalization bounds for deep convolutional neural networkscited by 59
  16. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 26
  17. 2019Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problemscited by 24
  18. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  19. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  20. 2021Learning with Gradient Descent and Weakly Convex Lossescited by 16
  21. 2022On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Modelcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 5.

9 papers.

  1. 2024Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methodscited by 7
  2. 2025Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processescited by 2
  3. 2026Optimization, Generalization and Differential Privacy Bounds for Gradient Descent on Kolmogorov-Arnold Networkscited by 1
  4. 2024How Does the Smoothness Approximation Method Facilitate Generalization for Federated Adversarial Learning?cited by 0
  5. 2025Generalization and Optimization of SGD with Lookaheadcited by 0
  6. 2026Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networkscited by 0
  7. 2026Generalization error bounds for two-layer neural networks with Lipschitz loss functioncited by 0
  8. 2026Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networkscited by 0
  9. 2026Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descentcited by 0