mlsanityData to 7 Oct 2026

Accelerated Linearized Laplace Approximation for Bayesian Deep Learning

arXiv 2210.12642 ↗First posted 23 Oct 2022cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Laplace approximation (LA) and its linearized variant (LLA) enable effortless adaptation of pretrained deep neural networks to Bayesian neural networks. The generalized Gauss-Newton (GGN) approximation is typically introduced to improve their tractability. However, LA and LLA are still confronted with non-trivial inefficiency issues and should rely on Kronecker-factored, diagonal, or even last-layer approximate GGN matrices in practical use. These approximations are likely to harm the fidelity of learning outcomes. To tackle this issue, inspired by the connections between LLA and neural tangent kernels (NTKs), we develop a Nystrom approximation to NTKs to accelerate LLA. Our method benefits from the capability of popular deep learning libraries for forward mode automatic differentiation, and enjoys reassuring theoretical guarantees. Extensive studies reflect the merits of the proposed method in aspects of both scalability and performance. Our method can even scale up to architectures like vision transformers. We also offer valuable ablation studies to diagnose our method. Code is available at urlhttps://github.com/thudzj/ELLA.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 34 of its 67 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

34 of 67 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  12. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
22 more
  1. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  2. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  4. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  5. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  6. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  7. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  8. 2019Practical Deep Learning with Bayesian Principlescited by 158
  9. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  10. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  11. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  12. 2016Orthogonal Random Featurescited by 89
  13. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  14. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 85
  15. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  16. 2019'In-Between' Uncertainty in Bayesian Neural Networkscited by 79
  17. 2019Rates of Convergence for Sparse Variational Gaussian Process Regressioncited by 62
  18. 2022NeuralEF: Deconstructing Kernels by Deep Neural Networkscited by 26
  19. 2018Quadrature-based features for kernel approximationcited by 25
  20. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  21. 2021Connections and Equivalences between the Nystr"om Method and Sparse Variational Gaussian Processescited by 17
  22. 2020BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 10; 2025: 5; 2026: 7.

27 papers.

Sort:

By area:

  1. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  2. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  3. 2023Variational Linearized Laplace Approximation for Bayesian Deep Learningcited by 17
  4. 2024Reparameterization invariance in approximate Bayesian inferencecited by 17
  5. 2025Active operator learning with predictive uncertainty quantification for partial differential equationscited by 12
  6. 2024Linearization Turns Neural Operators into Function-Valued Gaussian Processescited by 11
  7. 2024FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learningcited by 11
  8. 2025Uncertainty Quantification with the Empirical Neural Tangent Kernelcited by 11
  9. 2023Stochastic Marginal Likelihood Gradients using Neural Tangent Kernelscited by 10
  10. 2024Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptationcited by 6
  11. 2024A Framework for Strategic Discovery of Credible Neural Network Surrogate Models under Uncertaintycited by 5
  12. 2024Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscited by 3
15 more
  1. 2026Richer Bayesian Last Layers with Subsampled NTK Featurescited by 3
  2. 2024Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learningcited by 2
  3. 2025Low Rank Based Subspace Inference for the Laplace Approximation of Bayesian Neural Networkscited by 2
  4. 2025CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regressioncited by 2
  5. 2023Uncertainty quantification in neural network classifiers -- a local linear approachcited by 1
  6. 2024Gradients of Functions of Large Matricescited by 1
  7. 2024Uncertainties of Latent Representations in Computer Visioncited by 1
  8. 2026Scalable Linearized Laplace Approximation via Surrogate Neural Kernelcited by 1
  9. 2026Is the Last Layer Sufficient for Uncertainty Quantification?cited by 1
  10. 2023Be Bayesian by Attachments to Catch More Uncertaintycited by 0
  11. 2025PEdger++: Practical Edge Detection via Assembling Cross Informationcited by 0
  12. 2026Amortized Acquisition Optimization for Bayesian Optimization with Variational Mutual Informationcited by 0
  13. 2026Activation-Space Uncertainty Quantification for Pretrained Networkscited by 0
  14. 2026VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learningcited by 0
  15. 2026Soft Specialists: α-R'enyi Ensembles for Uncertainty-Aware LLM Post-Trainingcited by 0