# Variational Auto-Regressive Gaussian Processes for Continual Learning

arXiv: https://arxiv.org/abs/2006.05468 · HTML: https://mlsanity.com/paper/2006.05468/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2020.

## Built on by (most cited first, 23 of 23)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors](https://mlsanity.com/paper/2205.10279/) (2022)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [Conditioning Sparse Variational Gaussian Processes for Online Decision-making](https://mlsanity.com/paper/2110.15172/) (2021)
- [Memory-Based Dual Gaussian Processes for Sequential Learning](https://mlsanity.com/paper/2306.03566/) (2023)
- [Function-space Parameterization of Neural Networks for Sequential Learning](https://mlsanity.com/paper/2403.10929/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Efficient Parametric Approximations of Neural Network Function Space Distance](https://mlsanity.com/paper/2302.03519/) (2023)
- [On Sequential Bayesian Inference for Continual Learning](https://mlsanity.com/paper/2301.01828/) (2023)
- [Continual Learning From a Stream of APIs](https://mlsanity.com/paper/2309.00023/) (2023)
- [Black-box Coreset Variational Inference](https://mlsanity.com/paper/2211.02377/) (2022)
- [Bayesian Transfer Learning](https://mlsanity.com/paper/2312.13484/) (2023)
- [Recurrent Memory for Online Interdomain Gaussian Processes](https://mlsanity.com/paper/2502.08736/) (2025)
- [Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?](https://mlsanity.com/paper/2408.07588/) (2024)
- [Likelihood approximations via Gaussian approximate inference](https://mlsanity.com/paper/2410.20754/) (2024)
- [Generative Kernel Continual learning](https://mlsanity.com/paper/2112.13410/) (2021)
- [Hierarchical Memory Pool Based Edge Semi-Supervised Continual Learning Method](https://mlsanity.com/paper/2303.11952/) (2023)
- [Low-Rank Filtering and Smoothing for Sequential Deep Learning](https://mlsanity.com/paper/2410.06800/) (2024)
- [Multivariate and Online Transfer Learning with Uncertainty Quantification](https://mlsanity.com/paper/2411.12555/) (2024)
- [Monte Carlo Functional Regularisation for Continual Learning](https://mlsanity.com/paper/2508.13006/) (2025)

## Abstract (arXiv)

Through sequential construction of posteriors on observing data online, Bayes' theorem provides a natural framework for continual learning. We develop Variational Auto-Regressive Gaussian Processes (VAR-GPs), a principled posterior updating mechanism to solve sequential tasks in continual learning. By relying on sparse inducing point approximations for scalable posteriors, we propose a novel auto-regressive variational distribution which reveals two fruitful connections to existing results in Bayesian inference, expectation propagation and orthogonal inducing points. Mean predictive entropy estimates show VAR-GPs prevent catastrophic forgetting, which is empirically supported by strong performance on modern continual learning benchmarks against competitive baselines. A thorough ablation study demonstrates the efficacy of our modeling choices.