# Sparse Gaussian Process Hyperparameters: Optimize or Integrate?

arXiv: https://arxiv.org/abs/2211.02476 · HTML: https://mlsanity.com/paper/2211.02476/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [Doubly Stochastic Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/1705.08933/) (2017)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMC](https://mlsanity.com/paper/1903.08008/) (2019)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes](https://mlsanity.com/paper/1504.07027/) (2015)
- [Convergence of Sparse Variational Inference in Gaussian Processes Regression](https://mlsanity.com/paper/2008.00323/) (2020)
- [Hamiltonian Monte Carlo for Hierarchical Models](https://mlsanity.com/paper/1312.0906/) (2013)
- [A Framework for Interdomain and Multioutput Gaussian Processes](https://mlsanity.com/paper/2003.01115/) (2020)
- [MCMC for Variationally Sparse Gaussian Processes](https://mlsanity.com/paper/1506.04000/) (2015)
- [Convolutional Gaussian Processes](https://mlsanity.com/paper/1709.01894/) (2017)
- [Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1806.05490/) (2018)
- [Approximate Inference for Fully Bayesian Gaussian Process Regression](https://mlsanity.com/paper/1912.13440/) (2019)
- [Partitioned Variational Inference: A unified framework encompassing federated and continual learning](https://mlsanity.com/paper/1811.11206/) (2018)
- [Marginalised Gaussian Processes with Nested Sampling](https://mlsanity.com/paper/2010.16344/) (2020)
- [Pseudo-Marginal Bayesian Inference for Gaussian Processes](https://mlsanity.com/paper/1310.0740/) (2013)

## Built on by (most cited first, 4 of 4)

- [Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes](https://mlsanity.com/paper/2302.04534/) (2023)
- [Fully Bayesian Differential Gaussian Processes through Stochastic Differential Equations](https://mlsanity.com/paper/2408.06069/) (2024)
- [Multifidelity-Augmented Gaussian Process Inputs for Surrogate Modeling from Scarce Data](https://mlsanity.com/paper/2603.22050/) (2026)
- [Occam's Razor is Only as Sharp as Your ELBO](https://mlsanity.com/paper/2604.25984/) (2026)

## Abstract (arXiv)

The kernel function and its hyperparameters are the central model selection choice in a Gaussian proces (Rasmussen and Williams, 2006). Typically, the hyperparameters of the kernel are chosen by maximising the marginal likelihood, an approach known as Type-II maximum likelihood (ML-II). However, ML-II does not account for hyperparameter uncertainty, and it is well-known that this can lead to severely biased estimates and an underestimation of predictive uncertainty. While there are several works which employ a fully Bayesian characterisation of GPs, relatively few propose such approaches for the sparse GPs paradigm. In this work we propose an algorithm for sparse Gaussian process regression which leverages MCMC to sample from the hyperparameter posterior within the variational inducing point framework of Titsias (2009). This work is closely related to Hensman et al. (2015b) but side-steps the need to sample the inducing points, thereby significantly improving sampling efficiency in the Gaussian likelihood case. We compare this scheme against natural baselines in literature along with stochastic variational GPs (SVGPs) along with an extensive computational analysis.