# Riemannian Laplace Approximation with the Fisher Metric

arXiv: https://arxiv.org/abs/2311.02766 · HTML: https://mlsanity.com/paper/2311.02766/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 90 references that are in the map.
- First posted on arXiv 5 Nov 2023.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [Riemannian Score-Based Generative Modelling](https://mlsanity.com/paper/2202.02763/) (2022)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Expectation Propagation for approximate Bayesian inference](https://mlsanity.com/paper/1301.2294/) (2013)
- [Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis](https://mlsanity.com/paper/1806.03884/) (2018)
- [Continuous Hierarchical Representations with Poincar'e Variational Auto-Encoders](https://mlsanity.com/paper/1901.06033/) (2019)
- ['In-Between' Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1906.11537/) (2019)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [Amortized Variational Inference: When and Why?](https://mlsanity.com/paper/2307.11018/) (2023)
- [Pathfinder: Parallel quasi-Newton variational inference](https://mlsanity.com/paper/2108.03782/) (2021)
- [A Locally Adaptive Normal Distribution](https://mlsanity.com/paper/1606.02518/) (2016)
- [Accelerating Natural Gradient with Higher-Order Invariance](https://mlsanity.com/paper/1803.01273/) (2018)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Riemannian Laplace approximations for Bayesian neural networks](https://mlsanity.com/paper/2306.07158/) (2023)
- [Metric Gaussian Variational Inference](https://mlsanity.com/paper/1901.11033/) (2019)
- [Geometric variational inference](https://mlsanity.com/paper/2105.10470/) (2021)
- [Lagrangian Manifold Monte Carlo on Monge Patches](https://mlsanity.com/paper/2202.00755/) (2022)
- [Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics](https://mlsanity.com/paper/2303.05101/) (2023)
- [Laplace approximation and the natural gradient for Gaussian process regression with the heteroscedastic Student-t model](https://mlsanity.com/paper/1712.07437/) (2017)
- [Warped geometric information on the optimisation of Euclidean functions](https://mlsanity.com/paper/2308.08305/) (2023)

## Built on by (most cited first, 7 of 7)

- [Reparameterization invariance in approximate Bayesian inference](https://mlsanity.com/paper/2406.03334/) (2024)
- [What's Inside Your Diffusion Model? A Score-Based Riemannian Metric to Explore the Data Manifold](https://mlsanity.com/paper/2505.11128/) (2025)
- [Geodesic Slice Sampler for Multimodal Distributions with Strong Curvature](https://mlsanity.com/paper/2502.21190/) (2025)
- [Geometric Gaussian Approximations of Probability Distributions](https://mlsanity.com/paper/2507.00616/) (2025)
- [Tubular Riemannian Laplace Approximations for Bayesian Neural Networks](https://mlsanity.com/paper/2512.24381/) (2025)
- [Reducing Memorisation in Generative Models via Riemannian Bayesian Inference](https://mlsanity.com/paper/2602.00199/) (2026)
- [Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics](https://mlsanity.com/paper/2605.15459/) (2026)

## Abstract (arXiv)

Laplace's method approximates a target density with a Gaussian distribution at its mode. It is computationally efficient and asymptotically exact for Bayesian inference due to the Bernstein-von Mises theorem, but for complex targets and finite-data posteriors it is often too crude an approximation. A recent generalization of the Laplace Approximation transforms the Gaussian approximation according to a chosen Riemannian geometry providing a richer approximation family, while still retaining computational efficiency. However, as shown here, its properties depend heavily on the chosen metric, indeed the metric adopted in previous work results in approximations that are overly narrow as well as being biased even at the limit of infinite data. We correct this shortcoming by developing the approximation family further, deriving two alternative variants that are exact at the limit of infinite data, extending the theoretical analysis of the method, and demonstrating practical improvements in a range of experiments.