# Scalable Bayesian Learning with posteriors

arXiv: https://arxiv.org/abs/2406.00104 · HTML: https://mlsanity.com/paper/2406.00104/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 91 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 30 of 57)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [A Kronecker-factored approximate Fisher matrix for convolution layers](https://mlsanity.com/paper/1602.01407/) (2016)

## Built on by (most cited first, 13 of 13)

- [Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?](https://mlsanity.com/paper/2602.06500/) (2026)
- [Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction tasks](https://mlsanity.com/paper/2505.11412/) (2025)
- [On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2603.22030/) (2026)
- [Thermodynamic Bayesian Inference](https://mlsanity.com/paper/2410.01793/) (2024)
- [Towards E-Value Based Stopping Rules for Bayesian Deep Ensembles](https://mlsanity.com/paper/2604.18089/) (2026)
- [Scalable Bayesian Monte Carlo: fast uncertainty estimation beyond deep ensembles](https://mlsanity.com/paper/2505.13585/) (2025)
- [A Complete Decomposition of Stochastic Differential Equations](https://mlsanity.com/paper/2601.07834/) (2026)
- [Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation](https://mlsanity.com/paper/2602.15925/) (2026)
- [bde: A Python Package for Bayesian Deep Ensembles via MILE](https://mlsanity.com/paper/2605.14146/) (2026)
- [Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning](https://mlsanity.com/paper/2605.21765/) (2026)
- [Bayesian Meta-Learning with Expert Feedback for Task-Shift Adaptation through Causal Embeddings](https://mlsanity.com/paper/2602.19788/) (2026)
- [Guiding Posterior Exploration with Optimizer-Derived Geometry](https://mlsanity.com/paper/2607.25312/) (2026)
- [Quenched Ensemble Sampling](https://mlsanity.com/paper/2609.15894/) (2026)

## Abstract (arXiv)

Although theoretically compelling, Bayesian learning with modern machine learning models is computationally challenging since it requires approximating a high dimensional posterior distribution. In this work, we (i) introduce posteriors, an easily extensible PyTorch library hosting general-purpose implementations making Bayesian learning accessible and scalable to large data and parameter regimes; (ii) present a tempered framing of stochastic gradient Markov chain Monte Carlo, as implemented in posteriors, that transitions seamlessly into optimization and unveils a minor modification to deep ensembles to ensure they are asymptotically unbiased for the Bayesian posterior, and (iii) demonstrate and compare the utility of Bayesian approximations through experiments including an investigation into the cold posterior effect and applications with large language models.