mlsanityData to 14 Oct 2026

Being a Bit Frequentist Improves Bayesian Neural Networks

arXiv 2106.10065 ↗First posted 18 Jun 2021cs.LG stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this "OOD training" technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN training by exploring four different ways of incorporating OOD data into Bayesian inference. We show in extensive experiments that OOD-trained BNNs are competitive to recent frequentist baselines. This work thus provides strong baselines for future work in Bayesian UQ.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 38 of its 59 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

38 of 59 references.

References · 38

  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2016Wide Residual Networkscited by 3,070
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2016Concrete Problems in AI Safetycited by 1,642
  11. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
26 more
  1. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  2. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  3. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  4. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  5. 2015Weight Uncertainty in Neural Networkscited by 554
  6. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  7. 2018Deep Learning for Classical Japanese Literaturecited by 422
  8. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  9. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  10. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  11. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  12. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  13. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  14. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  15. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  16. 2019Practical Deep Learning with Bayesian Principlescited by 158
  17. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  18. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  19. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  20. 2018Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Networkcited by 69
  21. 2019Towards neural networks that provably know when they don't knowcited by 69
  22. 2020Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examplescited by 40
  23. 2020Learnable Uncertainty under Laplace Approximationscited by 29
  24. 2015Universum Prescription: Regularization using Unlabeled Datacited by 16
  25. 2021Bayesian OOD detection with aleatoric uncertainty and outlier exposurecited by 16
  26. 2020An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidencecited by 11

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 5; 2024: 3; 2025: 2.

15 papers.

Sort:

By area:

  1. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 125
  2. 2022CARD: Classification and Regression Diffusion Modelscited by 103
  3. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  4. 2023Uncertainty Estimation by Fisher Information-based Evidential Deep Learningcited by 58
  5. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  6. 2022Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networkscited by 14
  7. 2025Uncertainty-Aware Graph Neural Networks: A Multi-Hop Evidence Fusion Approachcited by 14
  8. 2023Promises and Pitfalls of the Linearized Laplace in Bayesian Optimizationcited by 10
  9. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  10. 2022Robustness to corruption in pre-trained Bayesian neural networkscited by 5
  11. 2024Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscited by 3
  12. 2025VIKING: Deep variational inference with stochastic projectionscited by 3
3 more
  1. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  2. 2023Be Bayesian by Attachments to Catch More Uncertaintycited by 0
  3. 2023Preventing Arbitrarily High Confidence on Far-Away Data in Point-Estimated Discriminative Neural Networkscited by 0