mlsanityData to 14 Oct 2026

Laplace Redux -- Effortless Bayesian Deep Learning

arXiv 2106.14806 ↗First posted 28 Jun 2021cs.LG stat.MLcited by 365

Not on the map. Why?

Abstract (arXiv)

Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection. The Laplace approximation (LA) is a classic, and arguably the simplest family of approximations for the intractable posteriors of deep neural networks. Yet, despite its simplicity, the LA is not as popular as alternatives like variational Bayes or deep ensembles. This may be due to assumptions that the LA is expensive due to the involved Hessian computation, that it is difficult to implement, or that it yields inferior results. In this work we show that these are misconceptions: we (i) review the range of variants of the LA including versions with minimal cost overhead; (ii) introduce "laplace", an easy-to-use software library for PyTorch offering user-friendly access to all major flavors of the LA; and (iii) demonstrate through extensive experiments that the LA is competitive with more popular alternatives in terms of performance, while excelling in terms of computational cost. We hope that this work will serve as a catalyst to a wider adoption of the LA in practical deep learning, including in domains where Bayesian approaches are not typically considered at the moment.

Key facts. Cited by 365 papers in the map (data through 14 Oct 2026). Builds on 54 of its 100 references that are in the map. First posted on arXiv 28 Jun 2021.

Builds on

54 of 100 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  11. 2016Wide Residual Networkscited by 3,070
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
38 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  9. 2015Weight Uncertainty in Neural Networkscited by 554
  10. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  11. 2014New insights and perspectives on the natural gradient methodcited by 443
  12. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  13. 2017Variational Continual Learningcited by 339
  14. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
  15. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  16. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 231
  17. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  18. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  19. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
  20. 2019Practical Deep Learning with Bayesian Principlescited by 158
  21. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 150
  22. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  23. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  24. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  25. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  26. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 106
  27. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  28. 2019Subspace Inference for Bayesian Deep Learningcited by 95
  29. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  30. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
  31. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 80
  32. 2019BackPACK: Packing more into backpropcited by 78
  33. 2017On Quadratic Penalties in Elastic Weight Consolidationcited by 55
  34. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 51
  35. 2015Bayesian Dark Knowledgecited by 45
  36. 2021Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networkscited by 32
  37. 2020Fast Predictive Uncertainty for Classification with Bayesian Deep Networkscited by 31
  38. 2020Estimating Model Uncertainty of Neural Networks in Sparse Information Formcited by 31

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 46; 2023: 62; 2024: 68; 2025: 89; 2026: 93.

365 papers.

Sort:

By area:

  1. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  2. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 125
  3. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  4. 2023Bayes' Rays: Uncertainty Quantification for Neural Radiance Fieldscited by 91
  5. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  6. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  7. 2024Variational Bayesian Last Layerscited by 69
  8. 2024A Survey on LoRA of Large Language Modelscited by 67
  9. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  10. 2023FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields using Fisher Informationcited by 64
  11. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  12. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
38 more
  1. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
  2. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  3. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  4. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 49
  5. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  6. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  7. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
  8. 2022On Uncertainty, Tempering, and Data Augmentation in Bayesian Classificationcited by 46
  9. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  10. 2024BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcutscited by 37
  11. 2023BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inferencecited by 35
  12. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  13. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  14. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  15. 2022Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantitiescited by 28
  16. 2022Accelerated Linearized Laplace Approximation for Bayesian Deep Learningcited by 28
  17. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  18. 2022NeuralEF: Deconstructing Kernels by Deep Neural Networkscited by 27
  19. 2023Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorialcited by 27
  20. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  21. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  22. 2022Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEscited by 25
  23. 2022BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networkscited by 23
  24. 2023Low-rank extended Kalman filtering for online learning of neural networks from streaming datacited by 23
  25. 2025Generative Uncertainty in Diffusion Modelscited by 23
  26. 2023FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregationcited by 22
  27. 2023Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetrycited by 21
  28. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  29. 2024Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Modelscited by 20
  30. 2024Diffusion Model Guided Sampling with Pixel-Wise Aleatoric Uncertainty Estimationcited by 20
  31. 2023Fixing Overconfidence in Dynamic Neural Networkscited by 19
  32. 2023Random-Set Neural Networks (RS-NN)cited by 19
  33. 2022Theoretical characterization of uncertainty in high-dimensional linear classificationcited by 18
  34. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  35. 2023Variational Linearized Laplace Approximation for Bayesian Deep Learningcited by 18
  36. 2025Probabilistic neural operators for functional uncertainty quantificationcited by 18
  37. 2025POp-GS: Next Best View in 3D-Gaussian Splatting with P-Optimalitycited by 18
  38. 2026Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrievalcited by 18

50 most cited of 365 All 365 (JSON)