# Functional Regularisation for Continual Learning with Gaussian Processes

arXiv: https://arxiv.org/abs/1901.11356 · HTML: https://mlsanity.com/paper/1901.11356/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2019.

## Built on by (most cited first, 30 of 123)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Wide Neural Networks Forget Less Catastrophically](https://mlsanity.com/paper/2110.11526/) (2021)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Model Zoo: A Growing "Brain" That Learns Continually](https://mlsanity.com/paper/2106.03027/) (2021)
- [A Closer Look at Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2203.17269/) (2022)
- [Powerpropagation: A sparsity inducing weight reparameterisation](https://mlsanity.com/paper/2110.00296/) (2021)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [Natural continual learning: success is a journey, not (just) a destination](https://mlsanity.com/paper/2106.08085/) (2021)
- [Posterior Meta-Replay for Continual Learning](https://mlsanity.com/paper/2103.01133/) (2021)
- [Same State, Different Task: Continual Reinforcement Learning without Interference](https://mlsanity.com/paper/2106.02940/) (2021)
- [Information-theoretic Online Memory Selection for Continual Learning](https://mlsanity.com/paper/2204.04763/) (2022)
- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)
- [Continually Learning Self-Supervised Representations with Projected Functional Regularization](https://mlsanity.com/paper/2112.15022/) (2021)
- [The Challenges of Continuous Self-Supervised Learning](https://mlsanity.com/paper/2203.12710/) (2022)
- [Online Adaptation of Language Models with a Memory of Amortized Contexts](https://mlsanity.com/paper/2403.04317/) (2024)
- [Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer](https://mlsanity.com/paper/2101.09536/) (2021)
- [Continual Learning via Sequential Function-Space Variational Inference](https://mlsanity.com/paper/2312.17210/) (2023)
- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [Kernel Continual Learning](https://mlsanity.com/paper/2107.05757/) (2021)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning](https://mlsanity.com/paper/2102.07868/) (2021)

## Abstract (arXiv)

We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network. This method, referred to as functional regularisation for Continual Learning, avoids forgetting a previous task by constructing and memorising an approximate posterior belief over the underlying task-specific function. To achieve this we rely on a Gaussian process obtained by treating the weights of the last layer of a neural network as random and Gaussian distributed. Then, the training algorithm sequentially encounters tasks and constructs posterior beliefs over the task-specific functions by using inducing point sparse Gaussian process methods. At each step a new task is first learnt and then a summary is constructed consisting of (i) inducing inputs -- a fixed-size subset of the task inputs selected such that it optimally represents the task -- and (ii) a posterior distribution over the function values at these inputs. This summary then regularises learning of future tasks, through Kullback-Leibler regularisation terms. Our method thus unites approaches focused on (pseudo-)rehearsal with those derived from a sequential Bayesian inference perspective in a principled way, leading to strong results on accepted benchmarks.