mlsanityData to 7 Oct 2026

Do Bayesian Neural Networks Need To Be Fully Stochastic?

arXiv 2211.06291 ↗First posted 11 Nov 2022cs.LG cs.AI stat.MLcited by 63

Not on the map. Why?

Abstract (arXiv)

We investigate the benefit of treating all the parameters in a Bayesian neural network stochastically and find compelling theoretical and empirical evidence that this standard construction may be unnecessary. To this end, we prove that expressive predictive distributions require only small amounts of stochasticity. In particular, partially stochastic networks with only n stochastic biases are universal probabilistic predictors for n-dimensional predictive problems. In empirical investigations, we find no systematic benefit of full stochasticity across four different inference modalities and eight datasets; partially stochastic networks can match and sometimes even outperform fully stochastic networks, despite their reduced memory costs.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 38 of its 69 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

38 of 69 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2016Wide Residual Networkscited by 3,057
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  9. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  10. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  11. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  12. 2021Unsolved Problems in ML Safetycited by 291
26 more
  1. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  2. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  3. 2019Universal Approximation with Deep Narrow Networkscited by 198
  4. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  5. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  6. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  7. 2019Functional Variational Bayesian Neural Networkscited by 127
  8. 2021Bayesian Neural Network Priors Revisitedcited by 121
  9. 2021Epistemic Neural Networkscited by 120
  10. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  11. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
  12. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  13. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
  14. 2020The Case for Bayesian Deep Learningcited by 68
  15. 2019Reliable training and estimation of variance networkscited by 63
  16. 2018Variational Implicit Processescited by 41
  17. 2019A Unifying Bayesian View of Continual Learningcited by 38
  18. 2019Benchmarking the Neural Linear Model for Regressioncited by 31
  19. 2021Dangers of Bayesian Model Averaging under Covariate Shiftcited by 31
  20. 2020Learnable Uncertainty under Laplace Approximationscited by 29
  21. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  22. 2022Wide Mean-Field Bayesian Neural Networks Ignore the Datacited by 22
  23. 2018Overpruning in Variational Bayesian Neural Networkscited by 21
  24. 2021Function-space Inference with Sparse Implicit Processescited by 13
  25. 2021Wide Mean-Field Variational Bayesian Neural Networks Ignore the Datacited by 7
  26. 2021Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 24; 2025: 21; 2026: 5.

63 papers.

Sort:

By area:

  1. 2024Variational Bayesian Last Layerscited by 67
  2. 2023Prediction-Oriented Bayesian Active Learningcited by 60
  3. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  4. 2025Generative Uncertainty in Diffusion Modelscited by 20
  5. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  6. 2025Probabilistic neural operators for functional uncertainty quantificationcited by 17
  7. 2023Riemannian Laplace approximations for Bayesian neural networkscited by 14
  8. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  9. 2024Making Better Use of Unlabelled Data in Bayesian Active Learningcited by 10
  10. 2023Incorporating Unlabelled Data into Bayesian Neural Networkscited by 9
  11. 2024Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networkscited by 8
  12. 2024Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensemblescited by 8
38 more
  1. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  2. 2024Partially Stochastic Infinitely Deep Bayesian Neural Networkscited by 6
  3. 2023Implicit Variational Inference for High-Dimensional Posteriorscited by 5
  4. 2024Can a Confident Prior Replace a Cold Posterior?cited by 5
  5. 2023On Feynman--Kac training of partial Bayesian neural networkscited by 4
  6. 2024FedSI: Federated Subnetwork Inference for Efficient Uncertainty Quantificationcited by 4
  7. 2024Graph Structure Learning with Interpretable Bayesian Neural Networkscited by 4
  8. 2025In-Context Parametric Inference: Point or Distribution Estimators?cited by 4
  9. 2023Detecting Toxic Flowcited by 3
  10. 2024Active Learning with Fully Bayesian Neural Networks for Discontinuous and Nonstationary Datacited by 3
  11. 2024Lightning UQ Box: A Comprehensive Framework for Uncertainty Quantification in Deep Learningcited by 3
  12. 2024Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Samplingcited by 3
  13. 2025VIKING: Deep variational inference with stochastic projectionscited by 3
  14. 2025Uncertainty Reasoning with Photonic Bayesian Machinescited by 3
  15. 2026Error Analysis of Neural-Network-Based Engressioncited by 3
  16. 2024Bayesian Active Learning for Censored Regressioncited by 2
  17. 2024Uncertainty Aware Tropical Cyclone Wind Speed Estimation from Satellite Datacited by 2
  18. 2024Scalable Vertical Federated Learning via Data Augmentation and Amortized Inferencecited by 2
  19. 2025Low Rank Based Subspace Inference for the Laplace Approximation of Bayesian Neural Networkscited by 2
  20. 2025EvidMTL: Evidential Multi-Task Learning for Uncertainty-Aware Semantic Surface Mapping from Monocular RGB Imagescited by 2
  21. 2025Diffusion differentiable resamplingcited by 2
  22. 2023Bootstrap Your Own Variancecited by 1
  23. 2024Exploring Diffusion Models' Corruption Stage in Few-Shot Fine-tuning and Mitigating with Bayesian Neural Networkscited by 1
  24. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  25. 2024A Bayesian Mixture Model of Temporal Point Processes with Determinantal Point Process Priorcited by 1
  26. 2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1
  27. 2025Optimizing Data Augmentation through Bayesian Model Selectioncited by 1
  28. 2025Scaling Laws for Uncertainty in Deep Learningcited by 1
  29. 2025Knowledge Distillation of Uncertainty using Deep Latent Factor Modelcited by 1
  30. 2026Laplace Approximation for Bayesian Tensor Network Kernel Machinescited by 1
  31. 2023Variational Inference on the Final-Layer Output of Neural Networkscited by 0
  32. 2023Gibbs Sampling the Posterior of Neural Networkscited by 0
  33. 2023Affine Invariant Ensemble Transform Methods to Improve Predictive Uncertainty in Neural Networkscited by 0
  34. 2023Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networkscited by 0
  35. 2023Be Bayesian by Attachments to Catch More Uncertaintycited by 0
  36. 2024Forecasting VIX using Bayesian Deep Learningcited by 0
  37. 2024Multivariate Bayesian Last Layer for Regression with Uncertainty Quantification and Decompositioncited by 0
  38. 2024Structured Partial Stochasticity in Bayesian Neural Networkscited by 0

50 most cited of 63 All 63 (JSON)