mlsanityData to 14 Oct 2026

Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees

arXiv 2111.01721 ↗First posted 2 Nov 2021stat.ML cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We formulate natural gradient variational inference (VI), expectation propagation (EP), and posterior linearisation (PL) as extensions of Newton's method for optimising the parameters of a Bayesian posterior distribution. This viewpoint explicitly casts inference algorithms under the framework of numerical optimisation. We show that common approximations to Newton's method from the optimisation literature, namely Gauss-Newton and quasi-Newton methods (e.g., the BFGS algorithm), are still valid under this 'Bayes-Newton' framework. This leads to a suite of novel algorithms which are guaranteed to result in positive semi-definite (PSD) covariance matrices, unlike standard VI and EP. Our unifying viewpoint provides new insights into the connections between various inference schemes. All the presented methods apply to any model with a Gaussian prior and non-conjugate likelihood, which we demonstrate with (sparse) Gaussian processes and state space models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 22 of its 68 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

22 of 68 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2014Scalable Variational Gaussian Process Classificationcited by 228
  4. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  5. 2021The Bayesian Learning Rulecited by 99
  6. 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
  7. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  8. 2015Probabilistic Numerics and Uncertainty in Computationscited by 60
  9. 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
  10. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  11. 2018Partitioned Variational Inference: A unified framework encompassing federated and continual learningcited by 29
  12. 2015Stochastic Expectation Propagationcited by 22
10 more
  1. 2020Fast Variational Learning in State-Space Gaussian Process Modelscited by 20
  2. 2019Stein's Lemma for the Reparameterization Trick with Exponential Family Mixturescited by 18
  3. 2018State Space Gaussian Processes with Non-Gaussian Likelihoodcited by 12
  4. 2020State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processescited by 10
  5. 2021Sparse Algorithms for Markovian Gaussian Processescited by 10
  6. 2019Variational Bayes on Manifoldscited by 9
  7. 2016Chained Gaussian Processescited by 7
  8. 2021Combining Pseudo-Point and State Space Approximations for Sum-Separable Gaussian Processescited by 7
  9. 2014Approximate Inference for Nonstationary Heteroscedastic Gaussian process Regressioncited by 6
  10. 2018Gaussian process classification using posterior linearisationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 4; 2025: 2; 2026: 1.

13 papers.

Sort:

  1. 2024BlackJAX: Composable Bayesian inference in JAXcited by 26
  2. 2022Markovian Gaussian Process Variational Autoencoderscited by 16
  3. 2023Variational Gaussian Process Diffusion Processescited by 11
  4. 2025Robust and Conjugate Spatio-Temporal Gaussian Processescited by 9
  5. 2024Physics-Informed Variational State-Space Gaussian Processescited by 7
  6. 2023Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Modelscited by 3
  7. 2023Short-term Prediction and Filtering of Solar Power Using State-Space Gaussian Processescited by 1
  8. 2023Efficient Sensor Placement from Regression with Sparse Gaussian Processes in Continuous and Discrete Spacescited by 1
  9. 2022A Look at Improving Robustness in Visual-inertial SLAM by Moment Matchingcited by 0
  10. 2024Fearless Stochasticity in Expectation Propagationcited by 0
  11. 2024Learning Coupled Subspaces for Multi-Condition Spike Datacited by 0
  12. 2025Recursive Entropic Variational Inference for Nonlinear State-Space Modelscited by 0
1 more
  1. 2026IMU-Free Body-Frame State Estimation with Sparse Scene Flow for Quadcopterscited by 0