# Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models

arXiv: https://arxiv.org/abs/2306.04201 · HTML: https://mlsanity.com/paper/2306.04201/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Expectation Propagation for approximate Bayesian inference](https://mlsanity.com/paper/1301.2294/) (2013)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [Bayesian Model Selection, the Marginal Likelihood, and Generalization](https://mlsanity.com/paper/2202.11678/) (2022)
- [Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models](https://mlsanity.com/paper/1803.09151/) (2018)
- [Stochastic Expectation Propagation](https://mlsanity.com/paper/1506.04132/) (2015)
- [Fast Variational Learning in State-Space Gaussian Process Models](https://mlsanity.com/paper/2007.04731/) (2020)
- [Dual Parameterization of Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2111.03412/) (2021)
- [Automatic Reparameterisation of Probabilistic Programs](https://mlsanity.com/paper/1906.03028/) (2019)
- [Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees](https://mlsanity.com/paper/2111.01721/) (2021)
- [Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models](https://mlsanity.com/paper/1412.7461/) (2014)

## Built on by (most cited first, 3 of 3)

- [Likelihood approximations via Gaussian approximate inference](https://mlsanity.com/paper/2410.20754/) (2024)
- [On the Laplace Approximation as Model Selection Criterion for Gaussian Processes](https://mlsanity.com/paper/2403.09215/) (2024)
- [Sparse Gaussian Processes: Structured Approximations and Power-EP Revisited](https://mlsanity.com/paper/2507.02377/) (2025)

## Abstract (arXiv)

Approximate inference in Gaussian process (GP) models with non-conjugate likelihoods gets entangled with the learning of the model hyperparameters. We improve hyperparameter learning in GP models and focus on the interplay between variational inference (VI) and the learning target. While VI's lower bound to the marginal likelihood is a suitable objective for inferring the approximate posterior, we show that a direct approximation of the marginal likelihood as in Expectation Propagation (EP) is a better learning objective for hyperparameter optimization. We design a hybrid training procedure to bring the best of both worlds: it leverages conjugate-computation VI for inference and uses an EP-like marginal likelihood approximation for hyperparameter learning. We compare VI, EP, Laplace approximation, and our proposed training procedure and empirically demonstrate the effectiveness of our proposal across a wide range of data sets.