# Riemannian Laplace approximations for Bayesian neural networks

arXiv: https://arxiv.org/abs/2306.07158 · HTML: https://mlsanity.com/paper/2306.07158/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [Practical Gauss-Newton Optimisation for Deep Learning](https://mlsanity.com/paper/1706.03662/) (2017)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [Approximate Inference Turns Deep Networks into Gaussian Processes](https://mlsanity.com/paper/1906.01930/) (2019)
- [Bayesian Deep Learning via Subnetwork Inference](https://mlsanity.com/paper/2010.14689/) (2020)
- ['In-Between' Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1906.11537/) (2019)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [Variational Boosting: Iteratively Refining Posterior Approximations](https://mlsanity.com/paper/1611.06585/) (2016)
- [Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning](https://mlsanity.com/paper/1907.00865/) (2019)
- [Adapting the Linearised Laplace Model Evidence for Modern Deep Learning](https://mlsanity.com/paper/2206.08900/) (2022)
- [Metrics for Probabilistic Geometries](https://mlsanity.com/paper/1411.7432/) (2014)
- [Fast and Robust Shortest Paths on Manifolds Learned from Data](https://mlsanity.com/paper/1901.07229/) (2019)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [A Locally Adaptive Normal Distribution](https://mlsanity.com/paper/1606.02518/) (2016)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks](https://mlsanity.com/paper/2205.10041/) (2022)
- [Probabilistic Solutions to Differential Equations and their Application to Riemannian Statistics](https://mlsanity.com/paper/1306.0308/) (2013)
- [Lagrangian Manifold Monte Carlo on Monge Patches](https://mlsanity.com/paper/2202.00755/) (2022)

## Built on by (most cited first, 15 of 15)

- [A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?](https://mlsanity.com/paper/2402.05015/) (2024)
- [Reparameterization invariance in approximate Bayesian inference](https://mlsanity.com/paper/2406.03334/) (2024)
- [The Interpolating Information Criterion for Overparameterized Models](https://mlsanity.com/paper/2307.07785/) (2023)
- [Riemannian Laplace Approximation with the Fisher Metric](https://mlsanity.com/paper/2311.02766/) (2023)
- [What's Inside Your Diffusion Model? A Score-Based Riemannian Metric to Explore the Data Manifold](https://mlsanity.com/paper/2505.11128/) (2025)
- [Geometric Gaussian Approximations of Probability Distributions](https://mlsanity.com/paper/2507.00616/) (2025)
- [Laplace Sample Information: Data Informativeness Through a Bayesian Lens](https://mlsanity.com/paper/2505.15303/) (2025)
- [Walking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networks](https://mlsanity.com/paper/2512.01500/) (2025)
- [Tubular Riemannian Laplace Approximations for Bayesian Neural Networks](https://mlsanity.com/paper/2512.24381/) (2025)
- [Reducing Memorisation in Generative Models via Riemannian Bayesian Inference](https://mlsanity.com/paper/2602.00199/) (2026)
- [From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network](https://mlsanity.com/paper/2512.13111/) (2025)
- [VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learning](https://mlsanity.com/paper/2604.08639/) (2026)
- [Monte Carlo Stochastic Depth for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/2604.12719/) (2026)
- [Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics](https://mlsanity.com/paper/2605.15459/) (2026)
- [Contrastive Linearized Laplace: Loss Geometry, Valid Repairs, and Reliable Cores](https://mlsanity.com/paper/2610.14875/) (2026)

## Abstract (arXiv)

Bayesian neural networks often approximate the weight-posterior with a Gaussian distribution. However, practical posteriors are often, even locally, highly non-Gaussian, and empirical performance deteriorates. We propose a simple parametric approximate posterior that adapts to the shape of the true posterior through a Riemannian metric that is determined by the log-posterior gradient. We develop a Riemannian Laplace approximation where samples naturally fall into weight-regions with low negative log-posterior. We show that these samples can be drawn by solving a system of ordinary differential equations, which can be done efficiently by leveraging the structure of the Riemannian metric and automatic differentiation. Empirically, we demonstrate that our approach consistently improves over the conventional Laplace approximation across tasks. We further show that, unlike the conventional Laplace approximation, our method is not overly sensitive to the choice of prior, which alleviates a practical pitfall of current approaches.