mlsanityData to 7 Oct 2026

Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks

arXiv 2602.22936 ↗First posted 26 Feb 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Algorithmic stability is among the most potent techniques in generalization analysis. However, its derivation usually requires a stepsize ηt = O(1/t) under non-convex training regimes, where t denotes iterations. This rigid decay of the stepsize potentially impedes optimization and may not align with practical scenarios. In this paper, we derive the generalization bounds under the homogeneous neural network regimes, proving that this regime enables slower stepsize decay of order Omega(1/√t) under mild assumptions. We further extend the theoretical results from several aspects, e.g., non-Lipschitz regimes. This finding is broadly applicable, as homogeneous neural networks encompass fully-connected and convolutional neural networks with ReLU and LeakyReLU activations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 66 of its 81 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

66 of 81 references.

References · 66

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  10. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  11. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  12. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
38 more
  1. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  2. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  3. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 183
  4. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  5. 2020Directional convergence and alignment in deep learningcited by 172
  6. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  7. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  8. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  9. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
  10. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 118
  11. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  12. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
  13. 2019Sharper bounds for uniformly stable algorithmscited by 91
  14. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 90
  15. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  16. 2021Private Stochastic Convex Optimization: Optimal Rates in ℓ1 Geometrycited by 83
  17. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  18. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  19. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  20. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  21. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  22. 2019On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learningcited by 57
  23. 2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
  24. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
  25. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  26. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  27. 2021Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settingscited by 45
  28. 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
  29. 2016Generalization of ERM in Stochastic Convex Optimization: The Dimension Strikes Backcited by 37
  30. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  31. 2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
  32. 2021Generalization bounds via distillationcited by 33
  33. 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
  34. 2022Generalization Bounds via Convex Analysiscited by 27
  35. 2022On the Generalization Mystery in Deep Learningcited by 25
  36. 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
  37. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  38. 2021SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochscited by 22

50 most cited of 66 All 66 (JSON)

Built on by

Not cited yet.