mlsanityData to 7 Oct 2026

Scalable Bayesian Learning with posteriors

arXiv 2406.00104 ↗First posted 31 May 2024cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Although theoretically compelling, Bayesian learning with modern machine learning models is computationally challenging since it requires approximating a high dimensional posterior distribution. In this work, we (i) introduce posteriors, an easily extensible PyTorch library hosting general-purpose implementations making Bayesian learning accessible and scalable to large data and parameter regimes; (ii) present a tempered framing of stochastic gradient Markov chain Monte Carlo, as implemented in posteriors, that transitions seamlessly into optimization and unveils a minor modification to deep ensembles to ensure they are asymptotically unbiased for the Bayesian posterior, and (iii) demonstrate and compare the utility of Bayesian approximations through experiments including an investigation into the cold posterior effect and applications with large language models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 57 of its 91 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

57 of 91 references.

References · 57

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2023Mistral 7Bcited by 3,065
  8. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  9. 2016Variational Inference: A Review for Statisticianscited by 1,466
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  12. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
38 more
  1. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  4. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  5. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  6. 2014New insights and perspectives on the natural gradient methodcited by 435
  7. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  8. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  9. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  10. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  11. 2017Variational Continual Learningcited by 335
  12. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  13. 2017Concrete Dropoutcited by 227
  14. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  15. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  16. 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 187
  17. 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
  18. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  19. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  20. 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
  21. 2021Epistemic Neural Networkscited by 120
  22. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  23. 2017Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inferencecited by 97
  24. 2015Measuring Sample Quality with Stein's Methodcited by 94
  25. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  26. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
  27. 2024Variational Bayesian Last Layerscited by 67
  28. 2019Stochastic gradient Markov chain Monte Carlocited by 56
  29. 2017On Quadratic Penalties in Elastic Weight Consolidationcited by 54
  30. 2020A statistical theory of cold posteriors in deep neural networkscited by 53
  31. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  32. 2020Bayesian Workflowcited by 47
  33. 2023Second-Order Uncertainty Quantification: A Distance-Based Approachcited by 35
  34. 2017Online Natural Gradient as a Kalman Filtercited by 25
  35. 2024Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rulescited by 21
  36. 2016Ensemble preconditioning for Markov chain Monte Carlo simulationcited by 16
  37. 2021Efficient Online Bayesian Inference for Neural Banditscited by 15
  38. 2023Fortuna: A Library for Uncertainty Quantification in Deep Learningcited by 14

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 10.

13 papers.

Sort:

  1. 2026Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?cited by 6
  2. 2025Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction taskscited by 3
  3. 2024Thermodynamic Bayesian Inferencecited by 2
  4. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 2
  5. 2026Towards E-Value Based Stopping Rules for Bayesian Deep Ensemblescited by 2
  6. 2026A Complete Decomposition of Stochastic Differential Equationscited by 1
  7. 2026Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisationcited by 1
  8. 2026bde: A Python Package for Bayesian Deep Ensembles via MILEcited by 1
  9. 2026Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learningcited by 1
  10. 2025Scalable Bayesian Monte Carlo: fast uncertainty estimation beyond deep ensemblescited by 0
  11. 2026Bayesian Meta-Learning with Expert Feedback for Task-Shift Adaptation through Causal Embeddingscited by 0
  12. 2026Guiding Posterior Exploration with Optimizer-Derived Geometrycited by 0
1 more
  1. 2026Quenched Ensemble Samplingcited by 0