mlsanityData to 7 Oct 2026

Generalisation under gradient descent via deterministic PAC-Bayes

arXiv 2209.02525 ↗First posted 6 Sep 2022stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows. Contrary to standard practice in the PAC-Bayesian setting, our result applies to optimisation algorithms that are deterministic, without requiring any de-randomisation step. Our bounds are fully computable, depending on the density of the initial distribution and the Hessian of the training objective over the trajectory. We show that our framework can be applied to a variety of iterative optimisation algorithms, including stochastic gradient descent (SGD), momentum-based schemes, and damped Hamiltonian dynamics.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 50 of its 81 references that are in the map. First posted on arXiv 6 Sep 2022.

Builds on

50 of 81 references.

References · 50

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  12. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
38 more
  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  3. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  4. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  5. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  6. 2019Linearized two-layers neural networks in high dimensioncited by 155
  7. 2019A Primer on PAC-Bayesian Learningcited by 130
  8. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  9. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  10. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  11. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
  12. 2020Tighter risk certificates for neural networkscited by 112
  13. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
  14. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  15. 2019Sharper bounds for uniformly stable algorithmscited by 91
  16. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  17. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  18. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  19. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 79
  20. 2020PAC-Bayes Analysis Beyond the Usual Boundscited by 63
  21. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  22. 2018Generalization Bounds for Uniformly Stable Algorithmscited by 56
  23. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  24. 2018Global Non-convex Optimization with Discretized Diffusionscited by 44
  25. 2020A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularizationcited by 44
  26. 2019Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networkscited by 38
  27. 2020Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networkscited by 29
  28. 2019Conformal Symplectic and Relativistic Optimizationcited by 28
  29. 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
  30. 2022Generalization Bounds via Convex Analysiscited by 27
  31. 2022Non-Vacuous Generalisation Bounds for Shallow Neural Networkscited by 22
  32. 2021Learning PAC-Bayes Priors for Probabilistic Neural Networkscited by 21
  33. 2021On Margins and Derandomisation in PAC-Bayescited by 19
  34. 2021Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Boundcited by 16
  35. 2022Chained Generalisation Boundscited by 10
  36. 2020De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictorscited by 7
  37. 2021Conditionally Gaussian PAC-Bayescited by 7
  38. 2019PAC-Bayesian Transportation Boundcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 5; 2026: 2.

9 papers.

  1. 2023Data-dependent Generalization Bounds via Variable-Size Compressibilitycited by 9
  2. 2025Implicit Regularisation in Diffusion Models: An Algorithm-Dependent Generalisation Analysiscited by 6
  3. 2025Generalization Bounds for Markov Algorithms through Entropy Flow Computationscited by 2
  4. 2025Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained modelscited by 2
  5. 2023A note on regularised NTK dynamics with an application to PAC-Bayesian trainingcited by 1
  6. 2025Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?cited by 1
  7. 2025Generalization of Gibbs and Langevin Monte Carlo Algorithms in the Interpolation Regimecited by 1
  8. 2026Smoothness-Based Derandomization of PAC-Bayes Boundscited by 1
  9. 2026Generalization at the Edge of Stabilitycited by 0