mlsanityData to 14 Oct 2026

Cold Posteriors through PAC-Bayes

arXiv 2206.11173 ↗First posted 22 Jun 2022cs.LG cs.AI stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

We investigate the cold posterior effect through the lens of PAC-Bayes generalization bounds. We argue that in the non-asymptotic setting, when the number of training samples is (relatively) small, discussions of the cold posterior effect should take into account that approximate Bayesian inference does not readily provide guarantees of performance on out-of-sample data. Instead, out-of-sample error is better described through a generalization bound. In this context, we explore the connections between the ELBO objective from variational inference and the PAC-Bayes objectives. We note that, while the ELBO and PAC-Bayes objectives are similar, the latter objectives naturally contain a temperature parameter λ which is not restricted to be λ=1. For both regression and classification tasks, in the case of isotropic Laplace approximations to the posterior, we show how this PAC-Bayesian interpretation of the temperature parameter captures the cold posterior effect.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

35 of 61 references.

References · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2016Wide Residual Networkscited by 3,070
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  7. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  8. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  9. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  10. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 210
  11. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  12. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
23 more
  1. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  2. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  3. 2021Bayesian Neural Network Priors Revisitedcited by 121
  4. 2014Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing Itcited by 95
  5. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  6. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 88
  7. 2019BackPACK: Packing more into backpropcited by 78
  8. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
  9. 2020A statistical theory of cold posteriors in deep neural networkscited by 54
  10. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  11. 2021Correlated Input-Dependent Label Noise in Large-Scale Image Classificationcited by 37
  12. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  13. 2018SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradientcited by 34
  14. 2021Fast Adaptation with Linearized Neural Networkscited by 32
  15. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  16. 2020Speedy Performance Estimation for Neural Architecture Searchcited by 24
  17. 2020Cold Posteriors and Aleatoric Uncertaintycited by 22
  18. 2020Predicting Training Time Without Trainingcited by 22
  19. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  20. 2020A Bayesian Perspective on Training Speed and Model Selectioncited by 19
  21. 2020PACm-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regimecited by 15
  22. 2022How Tempering Fixes Data Augmentation in Bayesian Neural Networkscited by 13
  23. 2021Bayesian Confidence Calibration for Epistemic Uncertainty Modellingcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2.

3 papers.

  1. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  2. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 5
  3. 2022Streaming PAC-Bayes Gaussian process regression with a performance guarantee for online decision makingcited by 0