# Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees

arXiv: https://arxiv.org/abs/2111.01721 · HTML: https://mlsanity.com/paper/2111.01721/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 68 references that are in the map.
- First posted on arXiv 2 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Approximate Inference Turns Deep Networks into Gaussian Processes](https://mlsanity.com/paper/1906.01930/) (2019)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [Probabilistic Numerics and Uncertainty in Computations](https://mlsanity.com/paper/1506.01326/) (2015)
- [Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models](https://mlsanity.com/paper/1803.09151/) (2018)
- [Handling the Positive-Definite Constraint in the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10060/) (2020)
- [Partitioned Variational Inference: A unified framework encompassing federated and continual learning](https://mlsanity.com/paper/1811.11206/) (2018)
- [Stochastic Expectation Propagation](https://mlsanity.com/paper/1506.04132/) (2015)
- [Fast Variational Learning in State-Space Gaussian Process Models](https://mlsanity.com/paper/2007.04731/) (2020)
- [Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures](https://mlsanity.com/paper/1910.13398/) (2019)
- [State Space Gaussian Processes with Non-Gaussian Likelihood](https://mlsanity.com/paper/1802.04846/) (2018)
- [State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processes](https://mlsanity.com/paper/2007.05994/) (2020)
- [Sparse Algorithms for Markovian Gaussian Processes](https://mlsanity.com/paper/2103.10710/) (2021)
- [Variational Bayes on Manifolds](https://mlsanity.com/paper/1908.03097/) (2019)
- [Chained Gaussian Processes](https://mlsanity.com/paper/1604.05263/) (2016)
- [Combining Pseudo-Point and State Space Approximations for Sum-Separable Gaussian Processes](https://mlsanity.com/paper/2106.10210/) (2021)
- [Approximate Inference for Nonstationary Heteroscedastic Gaussian process Regression](https://mlsanity.com/paper/1404.5443/) (2014)
- [Gaussian process classification using posterior linearisation](https://mlsanity.com/paper/1809.04967/) (2018)

## Built on by (most cited first, 13 of 13)

- [BlackJAX: Composable Bayesian inference in JAX](https://mlsanity.com/paper/2402.10797/) (2024)
- [Markovian Gaussian Process Variational Autoencoders](https://mlsanity.com/paper/2207.05543/) (2022)
- [Variational Gaussian Process Diffusion Processes](https://mlsanity.com/paper/2306.02066/) (2023)
- [Robust and Conjugate Spatio-Temporal Gaussian Processes](https://mlsanity.com/paper/2502.02450/) (2025)
- [Physics-Informed Variational State-Space Gaussian Processes](https://mlsanity.com/paper/2409.13876/) (2024)
- [Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models](https://mlsanity.com/paper/2306.04201/) (2023)
- [Short-term Prediction and Filtering of Solar Power Using State-Space Gaussian Processes](https://mlsanity.com/paper/2302.00388/) (2023)
- [Efficient Sensor Placement from Regression with Sparse Gaussian Processes in Continuous and Discrete Spaces](https://mlsanity.com/paper/2303.00028/) (2023)
- [A Look at Improving Robustness in Visual-inertial SLAM by Moment Matching](https://mlsanity.com/paper/2205.13821/) (2022)
- [Fearless Stochasticity in Expectation Propagation](https://mlsanity.com/paper/2406.01801/) (2024)
- [Learning Coupled Subspaces for Multi-Condition Spike Data](https://mlsanity.com/paper/2410.19153/) (2024)
- [Recursive Entropic Variational Inference for Nonlinear State-Space Models](https://mlsanity.com/paper/2511.15409/) (2025)
- [IMU-Free Body-Frame State Estimation with Sparse Scene Flow for Quadcopters](https://mlsanity.com/paper/2608.20891/) (2026)

## Abstract (arXiv)

We formulate natural gradient variational inference (VI), expectation propagation (EP), and posterior linearisation (PL) as extensions of Newton's method for optimising the parameters of a Bayesian posterior distribution. This viewpoint explicitly casts inference algorithms under the framework of numerical optimisation. We show that common approximations to Newton's method from the optimisation literature, namely Gauss-Newton and quasi-Newton methods (e.g., the BFGS algorithm), are still valid under this 'Bayes-Newton' framework. This leads to a suite of novel algorithms which are guaranteed to result in positive semi-definite (PSD) covariance matrices, unlike standard VI and EP. Our unifying viewpoint provides new insights into the connections between various inference schemes. All the presented methods apply to any model with a Gaussian prior and non-conjugate likelihood, which we demonstrate with (sparse) Gaussian processes and state space models.