mlsanityData to 14 Oct 2026

Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks

arXiv 2205.10041 ↗First posted 20 May 2022cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Monte Carlo (MC) integration is the de facto method for approximating the predictive distribution of Bayesian neural networks (BNNs). But, even with many MC samples, Gaussian-based BNNs could still yield bad predictive performance due to the posterior approximation's error. Meanwhile, alternatives to MC integration tend to be more expensive or biased. In this work, we experimentally show that the key to good MC-approximated predictive distributions is the quality of the approximate posterior itself. However, previous methods for obtaining accurate posterior approximations are expensive and non-trivial to implement. We, therefore, propose to refine Gaussian approximate posteriors with normalizing flows. When applied to last-layer BNNs, it yields a simple post hoc method for improving pre-existing parametric approximations. We show that the resulting posterior approximation is competitive with even the gold-standard full-batch Hamiltonian Monte Carlo.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 44 of its 74 references that are in the map. First posted on arXiv 20 May 2022.

Builds on

44 of 74 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2016Wide Residual Networkscited by 3,070
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  8. 2015Variational Inference with Normalizing Flowscited by 2,170
  9. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  11. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  12. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
32 more
  1. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  2. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  3. 2015Weight Uncertainty in Neural Networkscited by 554
  4. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  5. 2018Pyro: Deep Universal Probabilistic Programmingcited by 380
  6. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  7. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  8. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  9. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  10. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  11. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  12. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  13. 2019Practical Deep Learning with Bayesian Principlescited by 158
  14. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  15. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  16. 2019Functional Variational Bayesian Neural Networkscited by 127
  17. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  18. 2019Subspace Inference for Bayesian Deep Learningcited by 95
  19. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
  20. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  21. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 80
  22. 2019BackPACK: Packing more into backpropcited by 78
  23. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
  24. 2018Uncertainty in the Variational Information Bottleneckcited by 39
  25. 2016Variational Boosting: Iteratively Refining Posterior Approximationscited by 37
  26. 2020Fast Predictive Uncertainty for Classification with Bayesian Deep Networkscited by 31
  27. 2020Transforming Gaussian Processes With Normalizing Flowscited by 28
  28. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  29. 2013High-Dimensional Probability Estimation with Deep Density Modelscited by 17
  30. 2020Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimationcited by 16
  31. 2021Being a Bit Frequentist Improves Bayesian Neural Networkscited by 15
  32. 2020An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidencecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 3; 2026: 2.

14 papers.

Sort:

  1. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
  2. 2023Riemannian Laplace approximations for Bayesian neural networkscited by 15
  3. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  4. 2023Collapsed Inference for Bayesian Deep Learningcited by 6
  5. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  6. 2023Implicit Variational Inference for High-Dimensional Posteriorscited by 5
  7. 2025Martingale Posterior Neural Networks for Fast Sequential Decision Makingcited by 4
  8. 2024Trustworthy Personalized Bayesian Federated Learning via Posterior Fine-Tunecited by 1
  9. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  10. 2025Walking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networkscited by 1
  11. 2023Preventing Arbitrarily High Confidence on Far-Away Data in Point-Estimated Discriminative Neural Networkscited by 0
  12. 2025On Local Posterior Structure in Deep Ensemblescited by 0
2 more
  1. 2026Practical Deep Heteroskedastic Regressioncited by 0
  2. 2026Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantificationcited by 0