mlsanityData to 7 Oct 2026

Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates

arXiv 1911.02151 ↗First posted 6 Nov 2019stat.ML cs.IT cs.LGcited by 118

Not on the map. Why?

Abstract (arXiv)

In this work, we improve upon the stepwise analysis of noisy iterative learning algorithms initiated by Pensia, Jog, and Loh (2018) and recently extended by Bu, Zou, and Veeravalli (2019). Our main contributions are significantly improved mutual information bounds for Stochastic Gradient Langevin Dynamics via data-dependent estimates. Our approach is based on the variational characterization of mutual information and the use of data-dependent priors that forecast the mini-batch gradient based on a subset of the training samples. Our approach is broadly applicable within the information-theoretic framework of Russo and Zou (2015) and Xu and Raginsky (2017). Our bound can be tied to a measure of flatness of the empirical risk surface. As compared with other bounds that depend on the squared norms of gradients, empirical investigations show that the terms in our bounds are orders of magnitude smaller.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 30; 2023: 21; 2024: 15; 2025: 21; 2026: 8.

118 papers.

Sort:

By area:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  2. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  3. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  4. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  5. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  6. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  7. 2021PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classescited by 31
  8. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  9. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 29
  10. 2021PAC-Bayes Information Bottleneckcited by 28
  11. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  12. 2023Randomized Adversarial Training via Taylor Expansioncited by 28
38 more
  1. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  2. 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
  3. 2022Generalization Bounds via Convex Analysiscited by 27
  4. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 24
  5. 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
  6. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 21
  7. 2022Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GDcited by 19
  8. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 18
  9. 2022Understanding Generalization via Leave-One-Out Conditional Mutual Informationcited by 16
  10. 2023Information Theoretic Lower Bounds for Information Theoretic Upper Boundscited by 16
  11. 2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15
  12. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  13. 2023Generalization Bounds with Data-dependent Fractal Dimensionscited by 15
  14. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  15. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  16. 2021Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithmscited by 12
  17. 2023Time-Independent Information-Theoretic Generalization Bounds for SGLDcited by 12
  18. 2024Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorizationcited by 12
  19. 2021Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learningcited by 11
  20. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
  21. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  22. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  23. 2021An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learningcited by 10
  24. 2021Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learningcited by 10
  25. 2022Chained Generalisation Boundscited by 10
  26. 2021Disparity Between Batches as a Signal for Early Stoppingcited by 9
  27. 2022Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimalitycited by 9
  28. 2022Generalisation under gradient descent via deterministic PAC-Bayescited by 9
  29. 2023Data-dependent Generalization Bounds via Variable-Size Compressibilitycited by 9
  30. 2024Uniform Generalization Bounds on Data-Dependent Hypothesis Sets via PAC-Bayesian Theory on Random Setscited by 9
  31. 2022Stability Based Generalization Bounds for Exponential Family Langevin Dynamicscited by 8
  32. 2022On Leave-One-Out Conditional Mutual Information For Generalizationcited by 8
  33. 2023Model-agnostic Measure of Generalization Difficultycited by 7
  34. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  35. 2022Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergencecited by 6
  36. 2022On the Importance of Gradient Norm in PAC-Bayesian Boundscited by 6
  37. 2022On generalization bounds for deep networks based on loss surface implicit regularizationcited by 5
  38. 2022Information-Theoretic Analysis of Minimax Excess Riskcited by 5

50 most cited of 118 All 118 (JSON)