mlsanityData to 14 Oct 2026

Time-Independent Information-Theoretic Generalization Bounds for SGLD

arXiv 2311.01046 ↗First posted 2 Nov 2023cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

We provide novel information-theoretic generalization bounds for stochastic gradient Langevin dynamics (SGLD) under the assumptions of smoothness and dissipativity, which are widely used in sampling and non-convex optimization studies. Our bounds are time-independent and decay to zero as the sample size increases, regardless of the number of iterations and whether the step size is fixed. Unlike previous studies, we derive the generalization error bounds by focusing on the time evolution of the Kullback--Leibler divergence, which is related to the stability of datasets and is the upper bound of the mutual information between output parameters and an input dataset. Additionally, we establish the first information-theoretic generalization bound when the training and test loss are the same by showing that a loss function of SGLD is sub-exponential. This bound is also time-independent and removes the problematic step size dependence in existing work, leading to an improved excess risk bound by combining our analysis with the existing non-convex optimization error bounds.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

25 of 44 references.

References · 25

  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  2. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  3. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  4. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  5. 2019Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Sufficescited by 227
  6. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  7. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 148
  8. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  9. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  10. 2018Privacy Amplification by Iterationcited by 100
  11. 2017Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimizationcited by 97
  12. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
13 more
  1. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  2. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 71
  3. 2019On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learningcited by 57
  4. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  5. 2019Improved Bounds for Discretization of Langevin Diffusions: Near-Optimal Rates without Convexitycited by 39
  6. 2019Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimizationcited by 38
  7. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  8. 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
  9. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 22
  10. 2022Integral Probability Metrics PAC-Bayes Boundscited by 19
  11. 2023Information Theoretic Lower Bounds for Information Theoretic Upper Boundscited by 17
  12. 2022Improved Convergence Rate of Stochastic Gradient Langevin Dynamics with Variance Reduction and its Application to Optimizationcited by 16
  13. 2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 4.

12 papers.

  1. 2024Information-theoretic Generalization Analysis for Expected Calibration Errorcited by 10
  2. 2024Uniform Generalization Bounds on Data-Dependent Hypothesis Sets via PAC-Bayesian Theory on Random Setscited by 9
  3. 2024Generalization Bounds via Conditional f-Informationcited by 4
  4. 2025Generalization Bounds for Markov Algorithms through Entropy Flow Computationscited by 2
  5. 2025Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processescited by 2
  6. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  7. 2025Generalization of Gibbs and Langevin Monte Carlo Algorithms in the Interpolation Regimecited by 1
  8. 2025Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variablescited by 0
  9. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  10. 2026Supervised Learning as Lossy Compression: Characterizing Generalization and Sample Complexity via Finite Blocklength Analysiscited by 0
  11. 2026Pool-based Active Learning as Noisy Lossy Compression: Characterizing Label Complexity via Finite Blocklength Analysiscited by 0
  12. 2026Information-Theoretic Generalization Bounds for Sequential Decision Makingcited by 0