# Theoretical Understanding of the Information Flow on Continual Learning Performance

arXiv: https://arxiv.org/abs/2204.12010 · HTML: https://mlsanity.com/paper/2204.12010/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Layer-adaptive sparsity for the Magnitude-based Pruning](https://mlsanity.com/paper/2010.07611/) (2020)
- [Continual Learning with Node-Importance based Adaptive Group Sparse Regularization](https://mlsanity.com/paper/2003.13726/) (2020)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Continual Learning via Neural Pruning](https://mlsanity.com/paper/1903.04476/) (2019)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [Toward Understanding Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/1908.01091/) (2019)
- [SpaceNet: Make Free Space For Continual Learning](https://mlsanity.com/paper/2007.07617/) (2020)
- [Incremental Classifier Learning with Generative Adversarial Networks](https://mlsanity.com/paper/1802.00853/) (2018)
- [Learn-Prune-Share for Lifelong Learning](https://mlsanity.com/paper/2012.06956/) (2020)
- [Formalizing the Generalization-Forgetting Trade-off in Continual Learning](https://mlsanity.com/paper/2109.14035/) (2021)
- [SPACE: Structured Compression and Sharing of Representational Space for Continual Learning](https://mlsanity.com/paper/2001.08650/) (2020)

## Built on by (most cited first, 8 of 8)

- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [A Statistical Theory of Regularization-Based Continual Learning](https://mlsanity.com/paper/2406.06213/) (2024)
- [Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective](https://mlsanity.com/paper/2505.20970/) (2025)
- [On the Theory of Continual Learning with Gradient Descent for Neural Networks](https://mlsanity.com/paper/2510.05573/) (2025)
- [Investigating the Impact of Weight Sharing Decisions on Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2311.09506/) (2023)
- [Towards Explaining Deep Neural Network Compression Through a Probabilistic Latent Space](https://mlsanity.com/paper/2403.00155/) (2024)
- [Information Consistent Pruning: How to Efficiently Search for Sparse Networks?](https://mlsanity.com/paper/2501.15592/) (2025)
- [Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts](https://mlsanity.com/paper/2411.09199/) (2024)

## Abstract (arXiv)

Continual learning (CL) is a setting in which an agent has to learn from an incoming stream of data sequentially. CL performance evaluates the model's ability to continually learn and solve new problems with incremental available information over time while retaining previous knowledge. Despite the numerous previous solutions to bypass the catastrophic forgetting (CF) of previously seen tasks during the learning process, most of them still suffer significant forgetting, expensive memory cost, or lack of theoretical understanding of neural networks' conduct while learning new tasks. While the issue that CL performance degrades under different training regimes has been extensively studied empirically, insufficient attention has been paid from a theoretical angle. In this paper, we establish a probabilistic framework to analyze information flow through layers in networks for task sequences and its impact on learning performance. Our objective is to optimize the information preservation between layers while learning new tasks to manage task-specific knowledge passing throughout the layers while maintaining model performance on previous tasks. In particular, we study CL performance's relationship with information flow in the network to answer the question "How can knowledge of information flow between layers be used to alleviate CF?". Our analysis provides novel insights of information adaptation within the layers during the incremental task learning process. Through our experiments, we provide empirical evidence and practically highlight the performance improvement across multiple tasks.