# Facilitating Bayesian Continual Learning by Natural Gradients and Stein Gradients

arXiv: https://arxiv.org/abs/1904.10644 · HTML: https://mlsanity.com/paper/1904.10644/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2019.

## Built on by (most cited first, 4 of 4)

- [Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning](https://mlsanity.com/paper/2110.04593/) (2021)
- [Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detection](https://mlsanity.com/paper/2308.03300/) (2023)
- [The Bayesian Approach to Continual Learning: An Overview](https://mlsanity.com/paper/2507.08922/) (2025)
- [Revised Regularization for Efficient Continual Learning through Correlation-Based Parameter Update in Bayesian Neural Networks](https://mlsanity.com/paper/2411.14202/) (2024)

## Abstract (arXiv)

Continual learning aims to enable machine learning models to learn a general solution space for past and future tasks in a sequential manner. Conventional models tend to forget the knowledge of previous tasks while learning a new task, a phenomenon known as catastrophic forgetting. When using Bayesian models in continual learning, knowledge from previous tasks can be retained in two ways: 1). posterior distributions over the parameters, containing the knowledge gained from inference in previous tasks, which then serve as the priors for the following task; 2). coresets, containing knowledge of data distributions of previous tasks. Here, we show that Bayesian continual learning can be facilitated in terms of these two means through the use of natural gradients and Stein gradients respectively.