mlsanityData to 7 Oct 2026

Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization

arXiv 2304.08309 ↗First posted 17 Apr 2023cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. It is theoretically compelling since it can be seen as a Gaussian process posterior with the mean function given by the neural network's maximum-a-posteriori predictive function and the covariance function induced by the empirical neural tangent kernel. However, while its efficacy has been studied in large-scale tasks like image classification, it has not been studied in sequential decision-making problems like Bayesian optimization where Gaussian processes -- with simple mean functions and kernels such as the radial basis function -- are the de-facto surrogate models. In this work, we study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility. However, we also present some pitfalls that might arise and a potential problem with the LLA when the search space is unbounded.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 53 of its 88 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

53 of 88 references.

References · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2017Deep Neural Networks as Gaussian Processescited by 617
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  10. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  11. 2015Weight Uncertainty in Neural Networkscited by 554
  12. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
38 more
  1. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  2. 2014New insights and perspectives on the natural gradient methodcited by 435
  3. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  4. 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 336
  5. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  6. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  7. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  8. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  9. 2019Practical Deep Learning with Bayesian Principlescited by 158
  10. 2017Multiplicative Normalizing Flows for Variational Bayesian Neural Networkscited by 157
  11. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  12. 2021Bayesian Neural Network Priors Revisitedcited by 121
  13. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  14. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  15. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  16. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  17. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  18. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  19. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  20. 2019BackPACK: Packing more into backpropcited by 77
  21. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  22. 2019Function Space Particle Optimization for Bayesian Neural Networkscited by 41
  23. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  24. 2021Exact Langevin Dynamics with Stochastic Gradientscited by 30
  25. 2020Learnable Uncertainty under Laplace Approximationscited by 29
  26. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 28
  27. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  28. 2021Last Layer Marginal Likelihood for Invariance Learningcited by 21
  29. 2021On Stein Variational Neural Network Ensemblescited by 21
  30. 2021Periodic Activation Functions Induce Stationaritycited by 21
  31. 2021BNNpriors: A library for Bayesian neural network inference with different prior distributionscited by 20
  32. 2022Sampling-based inference for large linear models, with application to linearised Laplacecited by 18
  33. 2022Learning Invariant Weights in Neural Networkscited by 16
  34. 2021Being a Bit Frequentist Improves Bayesian Neural Networkscited by 15
  35. 2022Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networkscited by 14
  36. 2021Probing as Quantifying Inductive Biascited by 11
  37. 2020An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidencecited by 10
  38. 2022Laplacian Autoencoders for Learning Stochastic Representationscited by 10

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 4; 2026: 1.

10 papers.

  1. 2023A Study of Bayesian Neural Network Surrogates for Bayesian Optimizationcited by 50
  2. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  3. 2025Simplifying Bayesian Optimization Via In-Context Direct Optimum Samplingcited by 7
  4. 2023On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspectivecited by 6
  5. 2025laplax -- Laplace Approximations with JAXcited by 5
  6. 2026BayeSQP: Bayesian Optimization through Sequential Quadratic Programmingcited by 5
  7. 2024Uncertainty-Guided Likelihood Tree Searchcited by 4
  8. 2024How Useful is Intermittent, Asynchronous Expert Feedback for Bayesian Optimization?cited by 2
  9. 2025Bayesian Optimization for General Reaction Conditionscited by 0
  10. 2025Variational Deep Learning via Implicit Regularizationcited by 0