# PAC-Bayes with Backprop

arXiv: https://arxiv.org/abs/1908.07380 · HTML: https://mlsanity.com/paper/1908.07380/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2019.

## Built on by (most cited first, 30 of 36)

- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization](https://mlsanity.com/paper/2211.13609/) (2022)
- [Enhancing Adversarial Training with Second-Order Statistics of Weights](https://mlsanity.com/paper/2203.06020/) (2022)
- [Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability](https://mlsanity.com/paper/2102.06589/) (2021)
- [How Tight Can PAC-Bayes be in the Small Data Regime?](https://mlsanity.com/paper/2106.03542/) (2021)
- [PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingales](https://mlsanity.com/paper/2210.00928/) (2022)
- [Online PAC-Bayes Learning](https://mlsanity.com/paper/2206.00024/) (2022)
- [Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression](https://mlsanity.com/paper/2306.00788/) (2023)
- [More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validity](https://mlsanity.com/paper/2306.12214/) (2023)
- [PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descent](https://mlsanity.com/paper/2310.17588/) (2023)
- [PAC-Bayes-Chernoff bounds for unbounded losses](https://mlsanity.com/paper/2401.01148/) (2024)
- [FedHB: Hierarchical Bayesian Federated Learning](https://mlsanity.com/paper/2305.04979/) (2023)
- [PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparison](https://mlsanity.com/paper/2211.16110/) (2022)
- [Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Bounds](https://mlsanity.com/paper/2305.19243/) (2023)
- [PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction](https://mlsanity.com/paper/2312.04658/) (2023)
- [Task-Driven Detection of Distribution Shifts with Statistical Guarantees for Robot Learning](https://mlsanity.com/paper/2106.13703/) (2021)
- [Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss](https://mlsanity.com/paper/2405.14681/) (2024)
- [PAC Bayesian Performance Guarantees for Deep (Stochastic) Networks in Medical Imaging](https://mlsanity.com/paper/2104.05600/) (2021)
- [Wasserstein PAC-Bayes Learning: Exploiting Optimisation Guarantees to Explain Generalisation](https://mlsanity.com/paper/2304.07048/) (2023)
- [Federated Learning with Nonvacuous Generalisation Bounds](https://mlsanity.com/paper/2310.11203/) (2023)
- [Towards Understanding Task-agnostic Debiasing Through the Lenses of Intrinsic Bias and Forgetfulness](https://mlsanity.com/paper/2406.04146/) (2024)
- [Learning Provably Robust Motion Planners Using Funnel Libraries](https://mlsanity.com/paper/2111.08733/) (2021)
- [Stronger Generalization Guarantees for Robot Learning by Combining Generative Models and Real-World Data](https://mlsanity.com/paper/2111.08761/) (2021)
- [PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-loss](https://mlsanity.com/paper/2112.05547/) (2021)
- [Learning via Surrogate PAC-Bayes](https://mlsanity.com/paper/2410.10230/) (2024)
- [Some theoretical improvements on the tightness of PAC-Bayes risk certificates for neural networks](https://mlsanity.com/paper/2510.07935/) (2025)
- [Interactive Dynamic Walking: Learning Gait Switching Policies with Generalization Guarantees](https://mlsanity.com/paper/2109.13417/) (2021)
- [Streaming PAC-Bayes Gaussian process regression with a performance guarantee for online decision making](https://mlsanity.com/paper/2210.08486/) (2022)
- [Fundamental Tradeoffs in Learning with Prior Information](https://mlsanity.com/paper/2304.13479/) (2023)
- [A Hierarchical Bayesian Model for Deep Few-Shot Meta Learning](https://mlsanity.com/paper/2306.09702/) (2023)

## Abstract (arXiv)

We explore the family of methods "PAC-Bayes with Backprop" (PBB) to train probabilistic neural networks by minimizing PAC-Bayes bounds. We present two training objectives, one derived from a previously known PAC-Bayes bound, and a second one derived from a novel PAC-Bayes bound. Both training objectives are evaluated on MNIST and on various UCI data sets. Our experiments show two striking observations: we obtain competitive test set error estimates (~1.4% on MNIST) and at the same time we compute non-vacuous bounds with much tighter values (~2.3% on MNIST) than previous results. These observations suggest that neural nets trained by PBB may lead to self-bounding learning, where the available data can be used to simultaneously learn a predictor and certify its risk, with no need to follow a data-splitting protocol.