# Bayesian continual learning and forgetting in neural networks

arXiv: https://arxiv.org/abs/2504.13569 · HTML: https://mlsanity.com/paper/2504.13569/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 51 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Builds on (most cited first, 20 of 20)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Understanding Measures of Uncertainty for Adversarial Example Detection](https://mlsanity.com/paper/1803.08533/) (2018)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [A Unifying Bayesian View of Continual Learning](https://mlsanity.com/paper/1902.06494/) (2019)
- [Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates](https://mlsanity.com/paper/2010.00373/) (2020)
- [Synaptic metaplasticity in binarized neural networks](https://mlsanity.com/paper/2101.07592/) (2021)
- [VIBNN: Hardware Acceleration of Bayesian Neural Networks](https://mlsanity.com/paper/1802.00822/) (2018)
- [Statistical mechanics of continual learning: variational principle and mean-field potential](https://mlsanity.com/paper/2212.02846/) (2022)

## Built on by (most cited first, 9 of 9)

- [Learning to Evolve: Bayesian-Guided Continual Knowledge Graph Embedding](https://mlsanity.com/paper/2508.02426/) (2025)
- [Learning to Remember, Learn, and Forget in Attention-Based Models](https://mlsanity.com/paper/2602.09075/) (2026)
- [Intrinsic-Noise Consolidation: A Doob-Barrier-Conditioned Diffusion Turns Analog Device Noise into a Continual-Learning Resource](https://mlsanity.com/paper/2607.06924/) (2026)
- [Finding Structure in Continual Learning](https://mlsanity.com/paper/2602.04555/) (2026)
- [Task Switching Without Forgetting via Proximal Decoupling](https://mlsanity.com/paper/2604.18857/) (2026)
- [Active Continual Learning with Metaplastic Binary Bayesian Neural Networks](https://mlsanity.com/paper/2605.30198/) (2026)
- [A Noise Optimum in Rehearsal-Free Continual Learning: Isolation, Mechanism, and Scope](https://mlsanity.com/paper/2609.20162/) (2026)
- [The Sequential Price of Continual Learning](https://mlsanity.com/paper/2609.29674/) (2026)
- [Do Artificial Neural Networks Need Dendritic Nanotubes? Two Reasons for Shrinking Importance Estimates in Continual Learning](https://mlsanity.com/paper/2610.14258/) (2026)

## Abstract (arXiv)

Biological synapses effortlessly balance memory retention and flexibility, yet artificial neural networks still struggle with the extremes of catastrophic forgetting and catastrophic remembering. Here, we introduce Metaplasticity from Synaptic Uncertainty (MESU), a Bayesian framework that updates network parameters according their uncertainty. This approach allows a principled combination of learning and forgetting that ensures that critical knowledge is preserved while unused or outdated information is gradually released. Unlike standard Bayesian approaches -- which risk becoming overly constrained, and popular continual-learning methods that rely on explicit task boundaries, MESU seamlessly adapts to streaming data. It further provides reliable epistemic uncertainty estimates, allowing out-of-distribution detection, the only computational cost being to sample the weights multiple times to provide proper output statistics. Experiments on image-classification benchmarks demonstrate that MESU mitigates catastrophic forgetting, while maintaining plasticity for new tasks. When training 200 sequential permuted MNIST tasks, MESU outperforms established continual learning techniques in terms of accuracy, capability to learn additional tasks, and out-of-distribution data detection. Additionally, due to its non-reliance on task boundaries, MESU outperforms conventional learning techniques on the incremental training of CIFAR-100 tasks consistently in a wide range of scenarios. Our results unify ideas from metaplasticity, Bayesian inference, and Hessian-based regularization, offering a biologically-inspired pathway to robust, perpetual learning.