mlsanityData to 14 Oct 2026

On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

arXiv 2110.03128 ↗First posted 7 Oct 2021cs.LG cs.IT math.ITcited by 31

Not on the map. Why?

Abstract (arXiv)

This paper follows up on a recent work of Neu et al. (2021) and presents some new information-theoretic upper bounds for the generalization error of machine learning models, such as neural networks, trained with SGD. We apply these bounds to analyzing the generalization behaviour of linear and two-layer ReLU networks. Experimental study of these bounds provide some insights on the SGD training of neural networks. They also point to a new and simple regularization scheme which we show performs comparably to the current state of the art.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 58 of its 85 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

58 of 85 references.

References · 58

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  12. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
38 more
  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  4. 2017Exploring Generalization in Deep Learningcited by 600
  5. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  6. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  7. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  11. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  12. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  13. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  14. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  15. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  16. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  17. 2017Three Factors Influencing Minima in SGDcited by 273
  18. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  19. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 173
  20. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  21. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  22. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  23. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 119
  24. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  25. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  26. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 98
  27. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  28. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  29. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  30. 2015Generalization in Adaptive Data Analysis and Holdout Reusecited by 83
  31. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  32. 2019BackPACK: Packing more into backpropcited by 78
  33. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  34. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  35. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  36. 2020Do We Need Zero Training Loss After Achieving Zero Training Error?cited by 61
  37. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  38. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 11; 2025: 7; 2026: 2.

31 papers.

Sort:

  1. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  2. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  3. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 22
  4. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  5. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 19
  6. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  7. 2023Over-training with Mixup May Hurt Generalizationcited by 13
  8. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  9. 2023Time-Independent Information-Theoretic Generalization Bounds for SGLDcited by 12
  10. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  11. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  12. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 10
19 more
  1. 2024Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinementcited by 9
  2. 2023Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Boundscited by 6
  3. 2023Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspectivecited by 5
  4. 2024Generalization Bounds via Conditional f-Informationcited by 4
  5. 2024How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysiscited by 3
  6. 2024Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networkscited by 3
  7. 2025Fairness Overfitting in Machine Learning: An Information-Theoretic Perspectivecited by 3
  8. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  9. 2024Class-wise Generalization Error: an Information-Theoretic Analysiscited by 2
  10. 2025Information-Theoretic Generalization Bounds of Replay-based Continual Learningcited by 2
  11. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  12. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  13. 2024Sharpness-Aware Minimization for Evolutionary Feature Construction in Regressioncited by 1
  14. 2025VASSO: Variance Suppression for Sharpness-Aware Minimizationcited by 1
  15. 2024Error Bounds of Supervised Classification from Information-Theoretic Perspectivecited by 0
  16. 2025Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variablescited by 0
  17. 2025Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learningcited by 0
  18. 2025Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapescited by 0
  19. 2026Information-Theoretic Generalization Bounds for Sequential Decision Makingcited by 0