# Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI

arXiv: https://arxiv.org/abs/2402.00809 · HTML: https://mlsanity.com/paper/2402.00809/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 159 of its 239 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 30 of 159)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)

## Built on by (most cited first, 30 of 62)

- [How to Build the Virtual Cell with Artificial Intelligence: Priorities and Opportunities](https://mlsanity.com/paper/2409.11654/) (2024)
- [On Information-Theoretic Measures of Predictive Uncertainty](https://mlsanity.com/paper/2410.10786/) (2024)
- [Post-hoc Probabilistic Vision-Language Models](https://mlsanity.com/paper/2412.06014/) (2024)
- [Linearization Turns Neural Operators into Function-Valued Gaussian Processes](https://mlsanity.com/paper/2406.05072/) (2024)
- [Sources of Uncertainty in 3D Scene Reconstruction](https://mlsanity.com/paper/2409.06407/) (2024)
- [Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks](https://mlsanity.com/paper/2502.06335/) (2025)
- [Bayesian Kolmogorov Arnold Networks (Bayesian_KANs): A Probabilistic Approach to Enhance Accuracy and Interpretability](https://mlsanity.com/paper/2408.02706/) (2024)
- [Predictive Coding Networks and Inference Learning: Tutorial and Survey](https://mlsanity.com/paper/2407.04117/) (2024)
- [Physics-Informed Variational State-Space Gaussian Processes](https://mlsanity.com/paper/2409.13876/) (2024)
- [Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation](https://mlsanity.com/paper/2411.04358/) (2024)
- [A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation](https://mlsanity.com/paper/2411.16370/) (2024)
- [BLIPs: Bayesian Learned Interatomic Potentials](https://mlsanity.com/paper/2508.14022/) (2025)
- [Bayesian Sheaf Neural Networks](https://mlsanity.com/paper/2410.09590/) (2024)
- [Streamlining Prediction in Bayesian Deep Learning](https://mlsanity.com/paper/2411.18425/) (2024)
- [laplax -- Laplace Approximations with JAX](https://mlsanity.com/paper/2507.17013/) (2025)
- [From Global to Local: A Scalable Benchmark for Local Posterior Sampling](https://mlsanity.com/paper/2507.21449/) (2025)
- [SiLVR: Scalable Lidar-Visual Radiance Field Reconstruction with Uncertainty Quantification](https://mlsanity.com/paper/2502.02657/) (2025)
- [Flow Stochastic Segmentation Networks](https://mlsanity.com/paper/2507.18838/) (2025)
- [On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2603.22030/) (2026)
- [Position: agentic AI orchestration should be Bayes-consistent](https://mlsanity.com/paper/2605.00742/) (2026)
- [Uncertainty in Physics and AI: Taxonomy, Quantification, and Validation](https://mlsanity.com/paper/2605.10378/) (2026)
- [Thermodynamic Bayesian Inference](https://mlsanity.com/paper/2410.01793/) (2024)
- [Optimization Proxies using Limited Labeled Data and Training Time -- A Semi-Supervised Bayesian Neural Network Approach](https://mlsanity.com/paper/2410.03085/) (2024)
- [Why you don't overfit, and don't need Bayes if you only train for one epoch](https://mlsanity.com/paper/2411.14478/) (2024)
- [Bayesian Predictive Coding](https://mlsanity.com/paper/2503.24016/) (2025)
- [From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learning](https://mlsanity.com/paper/2504.15722/) (2025)
- [Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study](https://mlsanity.com/paper/2506.19418/) (2025)
- [Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems](https://mlsanity.com/paper/2411.14860/) (2024)
- [Explainable Bayesian deep learning through input-skip Latent Binary Bayesian Neural Networks](https://mlsanity.com/paper/2503.10496/) (2025)
- [Epistemic Errors of Imperfect Multitask Learners When Distributions Shift](https://mlsanity.com/paper/2505.23496/) (2025)

## Abstract (arXiv)

In the current landscape of deep learning research, there is a predominant emphasis on achieving high predictive accuracy in supervised tasks involving large image and language datasets. However, a broader perspective reveals a multitude of overlooked metrics, tasks, and data types, such as uncertainty, active and continual learning, and scientific data, that demand attention. Bayesian deep learning (BDL) constitutes a promising avenue, offering advantages across these diverse settings. This paper posits that BDL can elevate the capabilities of deep learning. It revisits the strengths of BDL, acknowledges existing challenges, and highlights some exciting research avenues aimed at addressing these obstacles. Looking ahead, the discussion focuses on possible ways to combine large-scale foundation models with BDL to unlock their full potential.