# A Theory for Knowledge Transfer in Continual Learning

arXiv: https://arxiv.org/abs/2208.06931 · HTML: https://mlsanity.com/paper/2208.06931/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 14 Aug 2022.

## Builds on (most cited first, 20 of 20)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Compacting, Picking and Growing for Unforgetting Continual Learning](https://mlsanity.com/paper/1910.06562/) (2019)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Optimal Continual Learning has Perfect Memory and is NP-hard](https://mlsanity.com/paper/2006.05188/) (2020)
- [An analytic theory of generalization dynamics and transfer learning in deep linear networks](https://mlsanity.com/paper/1809.10374/) (2018)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [A law of robustness for two-layers neural networks](https://mlsanity.com/paper/2009.14444/) (2020)
- [Using Task Descriptions in Lifelong Machine Learning for Improved Performance and Zero-Shot Transfer](https://mlsanity.com/paper/1710.03850/) (2017)
- [Regularization-based Continual Learning for Anomaly Detection in Discrete Manufacturing](https://mlsanity.com/paper/2101.00509/) (2021)
- [Lifelong Learning Metrics](https://mlsanity.com/paper/2201.08278/) (2022)
- [Simple Lifelong Learning Machines](https://mlsanity.com/paper/2004.12908/) (2020)
- [Towards a General Theory of Infinite-Width Limits of Neural Classifiers](https://mlsanity.com/paper/2003.05884/) (2020)

## Built on by (most cited first, 16 of 16)

- [Continual Learning: Applications and the Road Forward](https://mlsanity.com/paper/2311.11908/) (2023)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Continual Learning on Graphs: A Survey](https://mlsanity.com/paper/2402.06330/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Societal Alignment Frameworks Can Improve LLM Alignment](https://mlsanity.com/paper/2503.00069/) (2025)
- [On the Limitation and Experience Replay for GNNs in Continual Learning](https://mlsanity.com/paper/2302.03534/) (2023)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [Data-dependent and Oracle Bounds on Forgetting in Continual Learning](https://mlsanity.com/paper/2406.09370/) (2024)
- [Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learning](https://mlsanity.com/paper/2503.21419/) (2025)
- [Bridging Neuroscience and AI: Environmental Enrichment as a Model for Forward Knowledge Transfer](https://mlsanity.com/paper/2405.07295/) (2024)
- [The Cake that is Intelligence and Who Gets to Bake it: An AI Analogy and its Implications for Participation](https://mlsanity.com/paper/2502.03038/) (2025)
- [T2I-ConBench: Text-to-Image Benchmark for Continual Post-training](https://mlsanity.com/paper/2505.16875/) (2025)
- [Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2602.20796/) (2026)
- [Forgetting is Everywhere](https://mlsanity.com/paper/2511.04666/) (2025)
- [Continual Learning of Achieving Forgetting-free and Positive Knowledge Transfer](https://mlsanity.com/paper/2601.05623/) (2026)
- [Server-Proximal Aggregation for Federated Domain-Incremental Learning under Partial Participation: Task-Uniform Convergence and Backward Transfer](https://mlsanity.com/paper/2601.22274/) (2026)

## Abstract (arXiv)

Continual learning of a stream of tasks is an active area in deep neural networks. The main challenge investigated has been the phenomenon of catastrophic forgetting or interference of newly acquired knowledge with knowledge from previous tasks. Recent work has investigated forward knowledge transfer to new tasks. Backward transfer for improving knowledge gained during previous tasks has received much less attention. There is in general limited understanding of how knowledge transfer could aid tasks learned continually. We present a theory for knowledge transfer in continual supervised learning, which considers both forward and backward transfer. We aim at understanding their impact for increasingly knowledgeable learners. We derive error bounds for each of these transfer mechanisms. These bounds are agnostic to specific implementations (e.g. deep neural networks). We demonstrate that, for a continual learner that observes related tasks, both forward and backward transfer can contribute to an increasing performance as more tasks are observed.