mlsanityData to 14 Oct 2026

Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States

arXiv 2211.10691 ↗First posted 19 Nov 2022cs.LG cs.IT math.ITcited by 11

Not on the map. Why?

Abstract (arXiv)

Stochastic differential equations (SDEs) have been shown recently to characterize well the dynamics of training machine learning models with SGD. When the generalization error of the SDE approximation closely aligns with that of SGD in expectation, it provides two opportunities for understanding better the generalization behaviour of SGD through its SDE approximation. Firstly, viewing SGD as full-batch gradient descent with Gaussian gradient noise allows us to obtain trajectory-based generalization bound using the information-theoretic bound from Xu and Raginsky [2017]. Secondly, assuming mild conditions, we estimate the steady-state weight distribution of SDE and use information-theoretic bounds from Xu and Raginsky [2017] and Negrea et al. [2019] to establish terminal-state-based generalization bounds. Our proposed bounds have some advantages, notably the trajectory-based bound outperforms results in Wang and Mao [2022], and the terminal-state-based bound exhibits a fast decay rate comparable to stability-based bounds.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 64 of its 91 references that are in the map. First posted on arXiv 19 Nov 2022.

Builds on

64 of 91 references.

References · 64

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2017A Closer Look at Memorization in Deep Networkscited by 942
  6. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  7. 2017Exploring Generalization in Deep Learningcited by 600
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  11. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  12. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
38 more
  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  2. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  3. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  5. 2017Three Factors Influencing Minima in SGDcited by 273
  6. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  7. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  8. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  9. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  11. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  12. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  13. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 173
  14. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  15. 2020Implicit Gradient Regularizationcited by 146
  16. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  17. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  18. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  19. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  20. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  21. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  22. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  23. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  24. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  25. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  26. 2018A Walk with SGDcited by 78
  27. 2019BackPACK: Packing more into backpropcited by 78
  28. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  29. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  30. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  31. 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
  32. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  33. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 51
  34. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  35. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 44
  36. 2020Generalization Bounds via Information Density and Conditional Information Densitycited by 43
  37. 2019Where is the Information in a Deep Neural Network?cited by 42
  38. 2017Algorithmic stability and hypothesis complexitycited by 39

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 4; 2026: 1.

11 papers.

  1. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 22
  2. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
  3. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9
  4. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  5. 2023Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Boundscited by 6
  6. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  7. 2024Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspectivecited by 4
  8. 2024Generalization Bounds via Conditional f-Informationcited by 4
  9. 2025Towards Auto-Regressive Next-Token Prediction: In-Context Learning Emerges from Generalizationcited by 3
  10. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  11. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2