# On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes

arXiv: https://arxiv.org/abs/1504.07027 · HTML: https://mlsanity.com/paper/1504.07027/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2015.

## Built on by (most cited first, 30 of 57)

- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [Diffusion Generative Models in Infinite Dimensions](https://mlsanity.com/paper/2212.00886/) (2022)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)
- [The Gaussian Neural Process](https://mlsanity.com/paper/2101.03606/) (2021)
- [Inducing Point Allocation for Sparse Gaussian Processes in High-Throughput Bayesian Optimisation](https://mlsanity.com/paper/2301.10123/) (2023)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent](https://mlsanity.com/paper/2306.11589/) (2023)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [Meta-Learning Reliable Priors in the Function Space](https://mlsanity.com/paper/2106.03195/) (2021)
- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [Last Layer Marginal Likelihood for Invariance Learning](https://mlsanity.com/paper/2106.07512/) (2021)
- [Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning](https://mlsanity.com/paper/2205.06342/) (2022)
- [Improved Convergence Rates for Sparse Approximation Methods in Kernel-Based Learning](https://mlsanity.com/paper/2202.04005/) (2022)
- [Connections and Equivalences between the Nystr"om Method and Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2106.01121/) (2021)
- [Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR](https://mlsanity.com/paper/2303.13779/) (2023)
- [Variational Inference for Continuous-Time Switching Dynamical Systems](https://mlsanity.com/paper/2109.14492/) (2021)
- [Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling](https://mlsanity.com/paper/2206.07673/) (2022)
- [Approximation-Aware Bayesian Optimization](https://mlsanity.com/paper/2406.04308/) (2024)
- [Combining Pseudo-Point and State Space Approximations for Sum-Separable Gaussian Processes](https://mlsanity.com/paper/2106.10210/) (2021)
- [Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees](https://mlsanity.com/paper/2210.07893/) (2022)
- [FaIRGP: A Bayesian Energy Balance Model for Surface Temperatures Emulation](https://mlsanity.com/paper/2307.10052/) (2023)
- [NeuralSurv: Deep Survival Analysis with Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2505.11054/) (2025)
- [Sparse Gaussian Process Hyperparameters: Optimize or Integrate?](https://mlsanity.com/paper/2211.02476/) (2022)
- [On permutation symmetries in Bayesian neural network posteriors: a variational perspective](https://mlsanity.com/paper/2310.10171/) (2023)
- [Estimation of Riemannian distances between covariance operators and Gaussian processes](https://mlsanity.com/paper/2108.11683/) (2021)
- [Variational Gaussian Processes: A Functional Analysis View](https://mlsanity.com/paper/2110.12798/) (2021)
- [Integrated Variational Fourier Features for Fast Spatial Modelling with Gaussian Processes](https://mlsanity.com/paper/2308.14142/) (2023)
- [Tighter sparse variational Gaussian processes](https://mlsanity.com/paper/2502.04750/) (2025)
- [Active Learning in Gaussian Process State Space Model](https://mlsanity.com/paper/2108.00819/) (2021)
- [Modelling Non-Smooth Signals with Complex Spectral Structure](https://mlsanity.com/paper/2203.06997/) (2022)

## Abstract (arXiv)

The variational framework for learning inducing variables (Titsias, 2009a) has had a large impact on the Gaussian process literature. The framework may be interpreted as minimizing a rigorously defined Kullback-Leibler divergence between the approximating and posterior processes. To our knowledge this connection has thus far gone unremarked in the literature. In this paper we give a substantial generalization of the literature on this topic. We give a new proof of the result for infinite index sets which allows inducing points that are not data points and likelihoods that depend on all function values. We then discuss augmented index sets and show that, contrary to previous works, marginal consistency of augmentation is not enough to guarantee consistency of variational inference with the original model. We then characterize an extra condition where such a guarantee is obtainable. Finally we show how our framework sheds light on interdomain sparse approximations and sparse approximations for Cox processes.