mlsanityData to 7 Oct 2026

PAC-Bayes Analysis Beyond the Usual Bounds

arXiv 2006.13057 ↗First posted 23 Jun 2020stat.ML cs.LG stat.APcited by 63

Not on the map. Why?

Abstract (arXiv)

We focus on a stochastic learning model where the learner observes a finite set of training examples and the output of the learning process is a data-dependent distribution over a space of hypotheses. The learned data-dependent distribution is then used to make randomized predictions, and the high-level theme addressed here is guaranteeing the quality of predictions on examples that were not seen during training, i.e. generalization. In this setting the unknown quantity of interest is the expected risk of the data-dependent randomized predictor, for which upper bounds can be derived via a PAC-Bayes analysis, leading to PAC-Bayes bounds. Specifically, we present a basic PAC-Bayes inequality for stochastic kernels, from which one may derive extensions of various known PAC-Bayes bounds as well as novel bounds. We clarify the role of the requirements of fixed 'data-free' priors, bounded losses, and i.i.d. data. We highlight that those requirements were used to upper-bound an exponential moment term, while the basic PAC-Bayes theorem remains valid without those restrictions. We present three bounds that illustrate the use of data-dependent priors, including one for the unbounded square loss.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 14; 2023: 8; 2024: 11; 2025: 7; 2026: 10.

63 papers.

Sort:

By area:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  2. 2021Bayesian Neural Network Priors Revisitedcited by 121
  3. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  4. 2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 29
  5. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  6. 2023A unified recipe for deriving (time-uniform) PAC-Bayes boundscited by 25
  7. 2021How Tight Can PAC-Bayes be in the Small Data Regime?cited by 24
  8. 2022PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingalescited by 23
  9. 2022Online PAC-Bayes Learningcited by 22
  10. 2021Learning PAC-Bayes Priors for Probabilistic Neural Networkscited by 21
  11. 2022On PAC-Bayesian reconstruction guarantees for VAEscited by 20
  12. 2022Integral Probability Metrics PAC-Bayes Boundscited by 19
38 more
  1. 2021Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Boundcited by 16
  2. 2023Statistical Guarantees for Variational Autoencoders using PAC-Bayesian Theorycited by 16
  3. 2021Information Complexity and Generalization Boundscited by 14
  4. 2023PAC-Bayesian Generalization Bounds for Adversarial Generative Modelscited by 13
  5. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  6. 2021Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithmscited by 12
  7. 2024Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learningcited by 11
  8. 2021Progress in Self-Certified Neural Networkscited by 10
  9. 2022Generalisation under gradient descent via deterministic PAC-Bayescited by 9
  10. 2023Data-dependent Generalization Bounds via Variable-Size Compressibilitycited by 9
  11. 2024PAC-Bayes-Chernoff bounds for unbounded lossescited by 9
  12. 2024Uniform Generalization Bounds on Data-Dependent Hypothesis Sets via PAC-Bayesian Theory on Random Setscited by 9
  13. 2022PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparisoncited by 7
  14. 2024Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measurescited by 7
  15. 2021A unified PAC-Bayesian framework for machine unlearning via information risk minimizationcited by 6
  16. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  17. 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
  18. 2022Investigating Generalization by Controlling Normalized Margincited by 5
  19. 2023Comparing Comparators in Generalization Boundscited by 5
  20. 2024Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approachcited by 5
  21. 2022PAC-Bayesian Learning of Optimization Algorithmscited by 4
  22. 2024Better-than-KL PAC-Bayes Boundscited by 4
  23. 2024Generalization and Robustness of the Tilted Empirical Riskcited by 4
  24. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  25. 2022Formal limitations of sample-wise information-theoretic generalization boundscited by 3
  26. 2022A General framework for PAC-Bayes Bounds for Meta-Learningcited by 3
  27. 2023Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisationcited by 3
  28. 2025A Framework for Bounding Deterministic Risk with PAC-Bayes: Applications to Majority Votescited by 3
  29. 2024Tighter Generalisation Bounds via Interpolationcited by 2
  30. 2024Generalization of Hamiltonian algorithmscited by 2
  31. 2025Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priorscited by 2
  32. 2025Reconcile Certified Robustness and Accuracy for DNN-based Smoothed Majority Vote Classifiercited by 2
  33. 2021Kernel Interpolation as a Bayes Point Machinecited by 1
  34. 2023Practicality of generalization guarantees for unsupervised domain adaptation with neural networkscited by 1
  35. 2025Multiview Representation Learning via Distributed Joint Latent Space Structuringcited by 1
  36. 2025Generalization of Gibbs and Langevin Monte Carlo Algorithms in the Interpolation Regimecited by 1
  37. 2025PAC-Bayesian Bounds on Constrained f-Entropic Risk Measurescited by 1
  38. 2026Smoothness-Based Derandomization of PAC-Bayes Boundscited by 1

50 most cited of 63 All 63 (JSON)