# Dual Parameterization of Sparse Variational Gaussian Processes

arXiv: https://arxiv.org/abs/2111.03412 · HTML: https://mlsanity.com/paper/2111.03412/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 5 Nov 2021.

## Builds on (most cited first, 16 of 16)

- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Expectation Propagation for approximate Bayesian inference](https://mlsanity.com/paper/1301.2294/) (2013)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [A Framework for Interdomain and Multioutput Gaussian Processes](https://mlsanity.com/paper/2003.01115/) (2020)
- [Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models](https://mlsanity.com/paper/1803.09151/) (2018)
- [Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models](https://mlsanity.com/paper/1807.04489/) (2018)
- [Stochastic Expectation Propagation](https://mlsanity.com/paper/1506.04132/) (2015)
- [Variational Inference for Gaussian Process Models with Linear Complexity](https://mlsanity.com/paper/1711.10127/) (2017)
- [Fast Variational Learning in State-Space Gaussian Process Models](https://mlsanity.com/paper/2007.04731/) (2020)
- [Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions](https://mlsanity.com/paper/1511.00146/) (2015)
- [Orthogonally Decoupled Variational Gaussian Processes](https://mlsanity.com/paper/1809.08820/) (2018)
- [Sparse Algorithms for Markovian Gaussian Processes](https://mlsanity.com/paper/2103.10710/) (2021)

## Built on by (most cited first, 19 of 19)

- [Sampling-based inference for large linear models, with application to linearised Laplace](https://mlsanity.com/paper/2210.04994/) (2022)
- [Variational Gaussian Process Diffusion Processes](https://mlsanity.com/paper/2306.02066/) (2023)
- [Memory-Based Dual Gaussian Processes for Sequential Learning](https://mlsanity.com/paper/2306.03566/) (2023)
- [Modeling Latent Neural Dynamics with Gaussian Process Switching Linear Dynamical Systems](https://mlsanity.com/paper/2408.03330/) (2024)
- [Function-space Parameterization of Neural Networks for Sequential Learning](https://mlsanity.com/paper/2403.10929/) (2024)
- [Fantasizing with Dual GPs in Bayesian Optimization and Active Learning](https://mlsanity.com/paper/2211.01053/) (2022)
- [In-Context Learning of Stochastic Differential Equations with Foundation Inference Models](https://mlsanity.com/paper/2502.19049/) (2025)
- [Knowledge Adaptation as Posterior Correction](https://mlsanity.com/paper/2506.14262/) (2025)
- [Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees](https://mlsanity.com/paper/2210.07893/) (2022)
- [SING: SDE Inference via Natural Gradients](https://mlsanity.com/paper/2506.17796/) (2025)
- [Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models](https://mlsanity.com/paper/2306.04201/) (2023)
- [Federated ADMM from Bayesian Duality](https://mlsanity.com/paper/2506.13150/) (2025)
- [Optimization Guarantees for Square-Root Natural-Gradient Variational Inference](https://mlsanity.com/paper/2507.07853/) (2025)
- [Foundation Inference Models for Ordinary Differential Equations](https://mlsanity.com/paper/2602.08733/) (2026)
- [Sparse Function-space Representation of Neural Networks](https://mlsanity.com/paper/2309.02195/) (2023)
- [One-Shot Federated Learning with Bayesian Pseudocoresets](https://mlsanity.com/paper/2406.02177/) (2024)
- [Towards Improved Learning in Gaussian Processes: The Best of Two Worlds](https://mlsanity.com/paper/2211.06260/) (2022)
- [Inverse-Free Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2604.00697/) (2026)
- [A Generalization of Amari's Bayesian Duality](https://mlsanity.com/paper/2609.09126/) (2026)

## Abstract (arXiv)

Sparse variational Gaussian process (SVGP) methods are a common choice for non-conjugate Gaussian process inference because of their computational benefits. In this paper, we improve their computational efficiency by using a dual parameterization where each data example is assigned dual parameters, similarly to site parameters used in expectation propagation. Our dual parameterization speeds-up inference using natural gradient descent, and provides a tighter evidence lower bound for hyperparameter learning. The approach has the same memory cost as the current SVGP methods, but it is faster and more accurate.