# A Tutorial on Sparse Gaussian Processes and Variational Inference

arXiv: https://arxiv.org/abs/2012.13962 · HTML: https://mlsanity.com/paper/2012.13962/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2020.

## Built on by (most cited first, 30 of 31)

- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [Connections and Equivalences between the Nystr"om Method and Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2106.01121/) (2021)
- [Sparse Uncertainty Representation in Deep Learning with Inducing Weights](https://mlsanity.com/paper/2105.14594/) (2021)
- [Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization (ZoMBI)](https://mlsanity.com/paper/2208.13771/) (2022)
- [GPflux: A Library for Deep Gaussian Processes](https://mlsanity.com/paper/2104.05674/) (2021)
- [Data-Efficient Kernel Methods for Learning Differential Equations and Their Solution Operators: Algorithms and Error Analysis](https://mlsanity.com/paper/2503.01036/) (2025)
- [Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2402.01476/) (2024)
- [Graph Random Features for Scalable Gaussian Processes](https://mlsanity.com/paper/2509.03691/) (2025)
- [Gaussian Processes on Hypergraphs](https://mlsanity.com/paper/2106.01982/) (2021)
- [Variational sparse inverse Cholesky approximation for latent Gaussian processes via double Kullback-Leibler minimization](https://mlsanity.com/paper/2301.13303/) (2023)
- [A Black-Box Physics-Informed Estimator based on Gaussian Process Regression for Robot Inverse Dynamics Identification](https://mlsanity.com/paper/2310.06585/) (2023)
- [Variational Gaussian Processes: A Functional Analysis View](https://mlsanity.com/paper/2110.12798/) (2021)
- [Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow](https://mlsanity.com/paper/2103.14407/) (2021)
- [Nonnegative spatial factorization](https://mlsanity.com/paper/2110.06122/) (2021)
- [Amortized Variational Inference for Deep Gaussian Processes](https://mlsanity.com/paper/2409.12301/) (2024)
- [New Bounds for Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2502.08730/) (2025)
- [Recurrent Memory for Online Interdomain Gaussian Processes](https://mlsanity.com/paper/2502.08736/) (2025)
- [Variational Inference for Model-Free and Model-Based Reinforcement Learning](https://mlsanity.com/paper/2209.01693/) (2022)
- [Decreasing the Computing Time of Bayesian Optimization using Generalizable Memory Pruning](https://mlsanity.com/paper/2309.04510/) (2023)
- [Variational Bayesian surrogate modelling with application to robust design optimisation](https://mlsanity.com/paper/2404.14857/) (2024)
- [Towards scientific machine learning for granular material simulations -- challenges and opportunities](https://mlsanity.com/paper/2504.08766/) (2025)
- [A flexible Bayesian non-parametric mixture model reveals multiple dependencies of swap errors in visual working memory](https://mlsanity.com/paper/2505.01178/) (2025)
- [Flow-Induced Diagonal Gaussian Processes](https://mlsanity.com/paper/2509.17153/) (2025)
- [Hybrid Bayesian Neural Networks with Functional Probabilistic Layers](https://mlsanity.com/paper/2107.07014/) (2021)
- [AODisaggregation: toward global aerosol vertical profiles](https://mlsanity.com/paper/2205.04296/) (2022)
- [A Unified Approach to Multi-task Legged Navigation: Temporal Logic Meets Reinforcement Learning](https://mlsanity.com/paper/2407.06931/) (2024)
- [Gradient-based Sample Selection for Faster Bayesian Optimization](https://mlsanity.com/paper/2504.07742/) (2025)
- [No-Regret Gaussian Process Optimization of Time-Varying Functions](https://mlsanity.com/paper/2512.00517/) (2025)
- [Gradient-Enhanced Partitioned Gaussian Processes for Real-Time Quadrotor Dynamics Modeling](https://mlsanity.com/paper/2602.12487/) (2026)
- [An Analysis of Posterior Collapse, Parameterization and Initialization in Variational Deep Gaussian Processes](https://mlsanity.com/paper/2606.25882/) (2026)

## Abstract (arXiv)

Gaussian processes (GPs) provide a framework for Bayesian inference that can offer principled uncertainty estimates for a large range of problems. For example, if we consider regression problems with Gaussian likelihoods, a GP model enjoys a posterior in closed form. However, identifying the posterior GP scales cubically with the number of training examples and requires to store all examples in memory. In order to overcome these obstacles, sparse GPs have been proposed that approximate the true posterior GP with pseudo-training examples. Importantly, the number of pseudo-training examples is user-defined and enables control over computational and memory complexity. In the general case, sparse GPs do not enjoy closed-form solutions and one has to resort to approximate inference. In this context, a convenient choice for approximate inference is variational inference (VI), where the problem of Bayesian inference is cast as an optimization problem -- namely, to maximize a lower bound of the log marginal likelihood. This paves the way for a powerful and versatile framework, where pseudo-training examples are treated as optimization arguments of the approximate posterior that are jointly identified together with hyperparameters of the generative model (i.e. prior and likelihood). The framework can naturally handle a wide scope of supervised learning problems, ranging from regression with heteroscedastic and non-Gaussian likelihoods to classification problems with discrete labels, but also problems with multidimensional labels. The purpose of this tutorial is to provide access to the basic matter for readers without prior knowledge in both GPs and VI. A proper exposition to the subject enables also access to more recent advances (like importance-weighted VI as well as interdomain, multioutput and deep GPs) that can serve as an inspiration for new research ideas.