mlsanityData to 7 Oct 2026

Generalization Error Bounds for Deep Neural Networks Trained by SGD

arXiv 2206.03299 ↗First posted 7 Jun 2022cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Generalization error bounds for deep neural networks trained by stochastic gradient descent (SGD) are derived by combining a dynamical control of an appropriate parameter norm and the Rademacher complexity estimate based on parameter norms. The bounds explicitly depend on the loss along the training trajectory, and work for a wide range of network architectures including multilayer perceptron (MLP) and convolutional neural networks (CNN). Compared with other algorithm-depending generalization estimates such as uniform stability-based bounds, our bounds do not require L-smoothness of the nonconvex loss function, and apply directly to SGD instead of Stochastic Langevin gradient descent (SGLD). Numerical results show that our bounds are non-vacuous and robust with the change of optimizer and network hyperparameters.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

31 of 46 references.

References · 31

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  6. 2017Exploring Generalization in Deep Learningcited by 595
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  8. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  9. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  11. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  12. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
19 more
  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  3. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  4. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  5. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  6. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  7. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  8. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  9. 2019Sharper bounds for uniformly stable algorithmscited by 91
  10. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  11. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  12. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  13. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 70
  14. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  15. 2018On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyondcited by 24
  16. 2019Generalization Bounds for Convolutional Neural Networkscited by 21
  17. 2020Complexity Measures for Neural Networks with General Activation Functions Using Path-based Normscited by 19
  18. 2019A Priori Estimates of the Population Risk for Residual Networkscited by 14
  19. 2018Understanding Generalization and Optimization Performance of Deep CNNscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 5; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023Toward Understanding Generative Data Augmentationcited by 25
  2. 2024Generalization Bound and New Algorithm for Clean-Label Backdoor Attackcited by 11
  3. 2023More Communication Does Not Result in Smaller Generalization Error in Federated Learningcited by 10
  4. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 5
  5. 2024Differentiated Aggregation to Improve Generalization in Federated Learningcited by 4
  6. 2023On the rate of convergence of an over-parametrized Transformer classifier learned by gradient descentcited by 2
  7. 2025Generalization Bound for a General Class of Neural Ordinary Differential Equationscited by 1
  8. 2024Analysis of the rate of convergence of an over-parametrized convolutional neural network image classifier learned by gradient descentcited by 0
  9. 2024Generalization bounds for regression and classification on adaptive covering input domainscited by 0
  10. 2024Mechanisms of Generative Image-to-Image Translation Networkscited by 0
  11. 2025Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenoncited by 0
  12. 2025Generalizability of Neural Networks Minimizing Empirical Risk Based on Expressive Abilitycited by 0
2 more
  1. 2025Statistically guided deep learningcited by 0
  2. 2025Learning Guarantee of Reward Modeling Using Deep Neural Networkscited by 0