# Statistical mechanics of continual learning: variational principle and mean-field potential

arXiv: https://arxiv.org/abs/2212.02846 · HTML: https://mlsanity.com/paper/2212.02846/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 19 of 19)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization](https://mlsanity.com/paper/1802.01569/) (2018)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [A Unifying Bayesian View of Continual Learning](https://mlsanity.com/paper/1902.06494/) (2019)
- [Continual Learning with Adaptive Weights (CLAW)](https://mlsanity.com/paper/1911.09514/) (2019)
- [Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates](https://mlsanity.com/paper/2010.00373/) (2020)
- [Synaptic metaplasticity in binarized neural networks](https://mlsanity.com/paper/2101.07592/) (2021)
- [Phase Transitions in Transfer Learning for High-Dimensional Perceptrons](https://mlsanity.com/paper/2101.01918/) (2021)
- [Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks](https://mlsanity.com/paper/2105.07385/) (2021)
- [Origin of the computational hardness for learning with binary synapses](https://mlsanity.com/paper/1408.1784/) (2014)
- [Continual task learning in natural and artificial agents](https://mlsanity.com/paper/2210.04520/) (2022)
- [Algorithmic insights on continual learning from fruit flies](https://mlsanity.com/paper/2107.07617/) (2021)
- [Generalization in multitask deep neural classifiers: a statistical physics approach](https://mlsanity.com/paper/1910.13593/) (2019)
- [Learning credit assignment](https://mlsanity.com/paper/2001.03354/) (2020)
- [On the role of synaptic stochasticity in training low-precision neural networks](https://mlsanity.com/paper/1710.09825/) (2017)

## Built on by (most cited first, 9 of 9)

- [Disentangling and Mitigating the Impact of Task Similarity for Continual Learning](https://mlsanity.com/paper/2405.20236/) (2024)
- [Order parameters and phase transitions of continual learning in deep neural networks](https://mlsanity.com/paper/2407.10315/) (2024)
- [Optimal Task Order for Continual Learning of Multiple Tasks](https://mlsanity.com/paper/2502.03350/) (2025)
- [Bayesian continual learning and forgetting in neural networks](https://mlsanity.com/paper/2504.13569/) (2025)
- [Eight challenges in developing theory of intelligence](https://mlsanity.com/paper/2306.11232/) (2023)
- [Grokking as a Falsifiable Finite-Size Transition](https://mlsanity.com/paper/2603.24746/) (2026)
- [Dynamics of Meta-learning Representation in the Teacher-student Scenario](https://mlsanity.com/paper/2408.12545/) (2024)
- [Game Theory Meets Statistical Mechanics in Deep Learning Design](https://mlsanity.com/paper/2410.12264/) (2024)
- [Continual Learning as a Multiphase Moving-Boundary Problem](https://mlsanity.com/paper/2606.01863/) (2026)

## Abstract (arXiv)

An obstacle to artificial general intelligence is set by continual learning of multiple tasks of different nature. Recently, various heuristic tricks, both from machine learning and from neuroscience angles, were proposed, but they lack a unified theory ground. Here, we focus on continual learning in single-layered and multi-layered neural networks of binary weights. A variational Bayesian learning setting is thus proposed, where the neural networks are trained in a field-space, rather than gradient-ill-defined discrete-weight space, and furthermore, weight uncertainty is naturally incorporated, and modulates synaptic resources among tasks. From a physics perspective, we translate the variational continual learning into Franz-Parisi thermodynamic potential framework, where previous task knowledge acts as a prior and a reference as well. We thus interpret the continual learning of the binary perceptron in a teacher-student setting as a Franz-Parisi potential computation. The learning performance can then be analytically studied with mean-field order parameters, whose predictions coincide with numerical experiments using stochastic gradient descent methods. Based on the variational principle and Gaussian field approximation of internal preactivations in hidden layers, we also derive the learning algorithm considering weight uncertainty, which solves the continual learning with binary weights using multi-layered neural networks, and performs better than the currently available metaplasticity algorithm. Our proposed principled frameworks also connect to elastic weight consolidation, weight-uncertainty modulated learning, and neuroscience inspired metaplasticity, providing a theory-grounded method for the real-world multi-task learning with deep networks.