mlsanityData to 7 Oct 2026

Dangers of Bayesian Model Averaging under Covariate Shift

arXiv 2106.11905 ↗First posted 22 Jun 2021cs.LG stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data. However, Bayesian neural networks (BNNs) with high-fidelity approximate inference via full-batch Hamiltonian Monte Carlo achieve poor generalization under covariate shift, even underperforming classical estimation. We explain this surprising result, showing how a Bayesian model average can in fact be problematic under covariate shift, particularly in cases where linear dependencies in the input features cause a lack of posterior contraction. We additionally show why the same issue does not affect many approximate inference procedures, or classical maximum a-posteriori (MAP) training. Finally, we propose novel priors that improve the robustness of BNNs to many sources of covariate shift.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 37 of its 69 references that are in the map. First posted on arXiv 22 Jun 2021.

Builds on

37 of 69 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  12. 2016End to End Learning for Self-Driving Carscited by 1,000
25 more
  1. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  2. 2015Weight Uncertainty in Neural Networkscited by 554
  3. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  4. 2020What is being transferred in transfer learning?cited by 365
  5. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  6. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  7. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  8. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  9. 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
  10. 2021Bayesian Neural Network Priors Revisitedcited by 121
  11. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  12. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  13. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  14. 2018Neural Arithmetic Logic Unitscited by 69
  15. 2021Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditionscited by 68
  16. 2018Excessive Invariance Causes Adversarial Vulnerabilitycited by 66
  17. 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
  18. 2019Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Controlcited by 60
  19. 2021Out of Distribution Generalization in Machine Learningcited by 59
  20. 2020Robustness of Bayesian Neural Networks to Gradient-Based Attackscited by 48
  21. 2021Bayesian Inference with Certifiable Adversarial Robustnesscited by 26
  22. 2016Learning Scalable Deep Kernels with Recurrent Structurecited by 22
  23. 2020PACm-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regimecited by 15
  24. 2019Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learningcited by 11
  25. 2019Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Networkcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 9; 2023: 10; 2024: 10.

31 papers.

Sort:

By area:

  1. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  2. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  3. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 60
  4. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  5. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  6. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  7. 2023When are ensembles really effective?cited by 21
  8. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  9. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  10. 2023How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?cited by 14
  11. 2022Towards out of distribution generalization for problems in mechanicscited by 11
  12. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
19 more
  1. 2023Approximately Bayes-Optimal Pseudo Label Selectioncited by 10
  2. 2022Robust Bayesian Learning for Reliable Wireless AI: Framework and Applicationscited by 9
  3. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  4. 2022Tackling covariate shift with node-based Bayesian neural networkscited by 8
  5. 2023Input-gradient space particle inference for neural network ensemblescited by 7
  6. 2024Improving robustness to corruptions with multiplicative weight perturbationscited by 7
  7. 2021On out-of-distribution detection with Bayesian neural networkscited by 6
  8. 2022Robustness to corruption in pre-trained Bayesian neural networkscited by 5
  9. 2023Conformalizing Machine Translation Evaluationcited by 5
  10. 2024Can a Confident Prior Replace a Cold Posterior?cited by 5
  11. 2024BayTTA: Uncertainty-aware medical image classification with optimized test-time augmentation using Bayesian model averagingcited by 5
  12. 2023Perspectives on the State and Future of Deep Learning - 2023cited by 4
  13. 2024Learning to Explore for Stochastic Gradient MCMCcited by 4
  14. 2022Uncertainty Calibration in Bayesian Neural Networks via Distance-Aware Priorscited by 3
  15. 2022Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learningcited by 2
  16. 2023Uncertainty quantification in neural network classifiers -- a local linear approachcited by 1
  17. 2024Understanding and mitigating difficulties in posterior predictive evaluationcited by 1
  18. 2024Domain Generalization Guided by Large-Scale Pre-Trained Priorscited by 1
  19. 2024Addressing the Inconsistency in Bayesian Deep Learning via Generalized Laplace Approximationcited by 0