mlsanityData to 7 Oct 2026

Flat Seeking Bayesian Neural Networks

arXiv 2302.02713 ↗First posted 6 Feb 2023cs.LG cs.IT math.ITcited by 10

Not on the map. Why?

Abstract (arXiv)

Bayesian Neural Networks (BNNs) provide a probabilistic interpretation for deep learning models by imposing a prior distribution over model parameters and inferring a posterior distribution based on observed data. The model sampled from the posterior distribution can be used for providing ensemble predictions and quantifying prediction uncertainty. It is well-known that deep learning models with lower sharpness have better generalization ability. However, existing posterior inferences are not aware of sharpness/flatness in terms of formulation, possibly leading to high sharpness for the models sampled from them. In this paper, we develop theories, the Bayesian setting, and the variational inference approach for the sharpness-aware posterior. Specifically, the models sampled from our sharpness-aware posterior, and the optimal approximate posterior estimating this sharpness-aware posterior, have better flatness, hence possibly possessing higher generalization ability. We conduct experiments by leveraging the sharpness-aware posterior with state-of-the-art Bayesian Neural Networks, showing that the flat-seeking counterparts outperform their baselines in all metrics of interest.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

49 of 68 references.

References · 49

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2021Convex optimizationcited by 2,479
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2017Deep Mutual Learningcited by 621
  9. 2017Exploring Generalization in Deep Learningcited by 595
  10. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
37 more
  1. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  2. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  3. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  5. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  6. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  7. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  8. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  9. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  10. 2017Three Factors Influencing Minima in SGDcited by 270
  11. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  12. 2017Concrete Dropoutcited by 227
  13. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  14. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  15. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  16. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  17. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  18. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  19. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  20. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  21. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  22. 2022Sharpness-Aware Training for Freecited by 95
  23. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  24. 2020Relative Flatness and Generalizationcited by 88
  25. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  26. 2022When Do Flat Minima Optimizers Work?cited by 78
  27. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  28. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  29. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  30. 2022SAM as an Optimal Relaxation of Bayescited by 40
  31. 2018Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priorscited by 35
  32. 2017Gaussian variational approximation with a factor covariance structurecited by 22
  33. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  34. 2022Stochastic Multiple Target Sampling Gradient Descentcited by 14
  35. 2022Global-Local Regularization Via Distributional Robustnesscited by 11
  36. 2019Walsh-Hadamard Variational Inference for Bayesian Deep Learningcited by 8
  37. 2021Structured Dropout Variational Inference for Bayesian Neural Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 4; 2026: 2.

10 papers.

  1. 2024TASAR: Transfer-based Attack on Skeletal Action Recognitioncited by 7
  2. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  3. 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
  4. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  5. 2025Stochastic Weight Sharing for Bayesian Neural Networkscited by 2
  6. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  7. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  8. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  9. 2023Sharpness-Aware Teleportation on Riemannian Manifoldscited by 0
  10. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0