# Memory-Based Dual Gaussian Processes for Sequential Learning

arXiv: https://arxiv.org/abs/2306.03566 · HTML: https://mlsanity.com/paper/2306.03566/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Functional Regularisation for Continual Learning with Gaussian Processes](https://mlsanity.com/paper/1901.11356/) (2019)
- [Continual Deep Learning by Functional Regularisation of Memorable Past](https://mlsanity.com/paper/2004.14070/) (2020)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [The Parallel Knowledge Gradient Method for Batch Bayesian Optimization](https://mlsanity.com/paper/1606.04414/) (2016)
- [Convergence of Sparse Variational Inference in Gaussian Processes Regression](https://mlsanity.com/paper/2008.00323/) (2020)
- [Streaming Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1705.07131/) (2017)
- [Modeling and interpolation of the ambient magnetic field by Gaussian processes](https://mlsanity.com/paper/1509.04634/) (2015)
- [Variational Auto-Regressive Gaussian Processes for Continual Learning](https://mlsanity.com/paper/2006.05468/) (2020)
- [Conditioning Sparse Variational Gaussian Processes for Online Decision-making](https://mlsanity.com/paper/2110.15172/) (2021)
- [Knowledge-Adaptation Priors](https://mlsanity.com/paper/2106.08769/) (2021)
- [Dual Parameterization of Sparse Variational Gaussian Processes](https://mlsanity.com/paper/2111.03412/) (2021)
- [Fantasizing with Dual GPs in Bayesian Optimization and Active Learning](https://mlsanity.com/paper/2211.01053/) (2022)
- [BOSH: Bayesian Optimization by Sampling Hierarchically](https://mlsanity.com/paper/2007.00939/) (2020)

## Built on by (most cited first, 11 of 11)

- [Function-space Parameterization of Neural Networks for Sequential Learning](https://mlsanity.com/paper/2403.10929/) (2024)
- [Knowledge Adaptation as Posterior Correction](https://mlsanity.com/paper/2506.14262/) (2025)
- [Preventing Model Collapse in Gaussian Process Latent Variable Models](https://mlsanity.com/paper/2404.01697/) (2024)
- [Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling](https://mlsanity.com/paper/2410.04988/) (2024)
- [Recurrent Memory for Online Interdomain Gaussian Processes](https://mlsanity.com/paper/2502.08736/) (2025)
- [Sparse Function-space Representation of Neural Networks](https://mlsanity.com/paper/2309.02195/) (2023)
- [Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?](https://mlsanity.com/paper/2408.07588/) (2024)
- [Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery](https://mlsanity.com/paper/2607.29225/) (2026)
- [Sequential Causal Discovery with Noisy Language Model Priors](https://mlsanity.com/paper/2506.16234/) (2025)
- [Novel Pivoted Cholesky Decompositions for Efficient Gaussian Process Inference](https://mlsanity.com/paper/2507.20678/) (2025)
- [Joint Model and Data Sparsification via the Marginal Likelihood](https://mlsanity.com/paper/2605.29908/) (2026)

## Abstract (arXiv)

Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning. In such cases, errors can accumulate over time due to inaccuracies in the posterior, hyperparameters, and inducing points, making accurate learning challenging. Here, we present a method to keep all such errors in check using the recently proposed dual sparse variational GP. Our method enables accurate inference for generic likelihoods and improves learning by actively building and updating a memory of past data. We demonstrate its effectiveness in several applications involving Bayesian optimization, active learning, and continual learning.