mlsanityData to 7 Oct 2026

PAC-Bayes-Chernoff bounds for unbounded losses

arXiv 2401.01148 ↗First posted 2 Jan 2024stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce a new PAC-Bayes oracle bound for unbounded losses that extends Cram'er-Chernoff bounds to the PAC-Bayesian setting. The proof technique relies on controlling the tails of certain random variables involving the Cram'er transform of the loss. Our approach naturally leverages properties of Cram'er-Chernoff bounds, such as exact optimization of the free parameter in many PAC-Bayes bounds. We highlight several applications of the main theorem. Firstly, we show that our bound recovers and generalizes previous results. Additionally, our approach allows working with richer assumptions that result in more informative and potentially tighter bounds. In this direction, we provide a general bound under a new textitmodel-dependent assumption from which we obtain bounds based on parameter norms and log-Sobolev inequalities. Notably, many of these bounds can be minimized to obtain distributions beyond the Gibbs posterior and provide novel theoretical coverage to existing regularization techniques.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 41 of its 74 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

41 of 74 references.

References · 41

  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  4. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  5. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  6. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  7. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  8. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  9. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  10. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  11. 2019A Primer on PAC-Bayesian Learningcited by 130
  12. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
29 more
  1. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  2. 2020PAC-Bayes Analysis Beyond the Usual Boundscited by 63
  3. 2023Generalization Bounds: Perspectives from Information Theory and PAC-Bayescited by 60
  4. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  5. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  6. 2016Simpler PAC-Bayesian Bounds for Hostile Datacited by 45
  7. 2019Scaleable input gradient regularization for adversarial robustnesscited by 45
  8. 2020Generalization Bounds via Information Density and Conditional Information Densitycited by 43
  9. 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
  10. 2019PAC-Bayes with Backpropcited by 36
  11. 2020Second Order PAC-Bayesian Bounds for the Weighted Majority Votecited by 34
  12. 2021Diversity and Generalization in Neural Network Ensemblescited by 32
  13. 2017Gradient Regularization Improves Accuracy of Discriminative Modelscited by 26
  14. 2023A unified recipe for deriving (time-uniform) PAC-Bayes boundscited by 25
  15. 2018Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Boundscited by 24
  16. 2021How Tight Can PAC-Bayes be in the Small Data Regime?cited by 24
  17. 2022PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingalescited by 23
  18. 2023More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validitycited by 17
  19. 2019PAC-Bayes under potentially heavy tailscited by 16
  20. 2019Still no free lunches: the price to pay for tighter PAC-Bayes boundscited by 14
  21. 2021Information Complexity and Generalization Boundscited by 14
  22. 2021Chebyshev-Cantelli PAC-Bayes-Bennett Inequality for the Weighted Majority Votecited by 13
  23. 2022Split-kl and PAC-Bayes-split-kl Inequalities for Ternary Random Variablescited by 11
  24. 2022On the Importance of Gradient Norm in PAC-Bayesian Boundscited by 6
  25. 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
  26. 2023Comparing Comparators in Generalization Boundscited by 5
  27. 2020Upper and Lower Bounds on the Performance of Kernel PCAcited by 3
  28. 2023PAC-Chernoff Bounds: Understanding Generalization in the Interpolation Regimecited by 2
  29. 2016An Oracle Inequality for Quasi-Bayesian Non-Negative Matrix Factorizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 6.

9 papers.

  1. 2024Sample Compression Unleashed: New Generalization Bounds for Real Valued Lossescited by 4
  2. 2025PAC-Bayes Bounds for Multivariate Linear Regression and Linear Autoencoderscited by 3
  3. 2026Distributionally Robust PAC-Bayesian Controlcited by 1
  4. 2026Smoothness-Based Derandomization of PAC-Bayes Boundscited by 1
  5. 2024How Learning Dynamics Drive Adversarially Robust Generalization?cited by 0
  6. 2026Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networkscited by 0
  7. 2026A PAC-Bayesian View of Generalisation for Physics-Informed Machine Learningcited by 0
  8. 2026PAC-Bayesian Certificates for Quadratic Closed-Loop Controlcited by 0
  9. 2026How Far Can Sharpness and Complexity Jointly Explain Generalization?cited by 0