mlsanityData to 14 Oct 2026

Flat Posterior Does Matter For Bayesian Model Averaging

arXiv 2406.15664 ↗First posted 21 Jun 2024stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Bayesian neural networks (BNNs) estimate the posterior distribution of model parameters and utilize posterior samples for Bayesian Model Averaging (BMA) in prediction. However, despite the crucial role of flatness in the loss landscape in improving the generalization of neural networks, its impact on BMA has been largely overlooked. In this work, we explore how posterior flatness influences BMA generalization and empirically demonstrate that (1) most approximate Bayesian inference methods fail to yield a flat posterior and (2) BMA predictions, without considering posterior flatness, are less effective at improving generalization. To address this, we propose Flat Posterior-aware Bayesian Model Averaging (FP-BMA), a novel training objective that explicitly encourages flat posteriors in a principled Bayesian manner. We also introduce a Flat Posterior-aware Bayesian Transfer Learning scheme that enhances generalization in downstream tasks. Empirically, we show that FP-BMA successfully captures flat posteriors, improving generalization performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 56 of its 87 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

56 of 87 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
38 more
  1. 2019Natural Adversarial Examplescited by 1,408
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  7. 2017Exploring Generalization in Deep Learningcited by 600
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  9. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  10. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  11. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  12. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  13. 2013Black Box Variational Inferencecited by 319
  14. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  15. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  16. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  17. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  18. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  19. 2019Subspace Inference for Bayesian Deep Learningcited by 95
  20. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  21. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
  22. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  23. 2024Variational Bayesian Last Layerscited by 69
  24. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  25. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  26. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  27. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  28. 2022SAM as an Optimal Relaxation of Bayescited by 41
  29. 2016Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemescited by 40
  30. 2015Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescited by 39
  31. 2019Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayescited by 32
  32. 2015Bayesian model averaging: A systematic review and conceptual classificationcited by 29
  33. 2023Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspectivecited by 28
  34. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  35. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  36. 2021Being a Bit Frequentist Improves Bayesian Neural Networkscited by 15
  37. 2022Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networkscited by 14
  38. 2023Flat Seeking Bayesian Neural Networkscited by 10

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0