# Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective

arXiv: https://arxiv.org/abs/2505.20970 · HTML: https://mlsanity.com/paper/2505.20970/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [How catastrophic can catastrophic forgetting be in linear regression?](https://mlsanity.com/paper/2205.09588/) (2022)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Probing Representation Forgetting in Supervised and Unsupervised Continual Learning](https://mlsanity.com/paper/2203.13381/) (2022)
- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [A PAC-Bayesian bound for Lifelong Learning](https://mlsanity.com/paper/1311.2838/) (2013)
- [Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent](https://mlsanity.com/paper/2006.11942/) (2020)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [A Statistical Theory of Regularization-Based Continual Learning](https://mlsanity.com/paper/2406.06213/) (2024)
- [Understanding Forgetting in Continual Learning with Linear Regression](https://mlsanity.com/paper/2405.17583/) (2024)
- [Fixed Design Analysis of Regularization-Based Continual Learning](https://mlsanity.com/paper/2303.10263/) (2023)
- [Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis](https://mlsanity.com/paper/2001.03371/) (2020)
- [Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks](https://mlsanity.com/paper/2105.07385/) (2021)
- [Investigating Forgetting in Pre-Trained Representations Through Continual Learning](https://mlsanity.com/paper/2305.05968/) (2023)
- [Memory Bounds for Continual Learning](https://mlsanity.com/paper/2204.10830/) (2022)
- [Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting](https://mlsanity.com/paper/2304.00933/) (2023)
- [Provable Lifelong Learning of Representations](https://mlsanity.com/paper/2110.14098/) (2021)
- [Provable Continual Learning via Sketched Jacobian Approximations](https://mlsanity.com/paper/2112.05095/) (2021)
- [Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgetting](https://mlsanity.com/paper/2112.01653/) (2021)

## Built on by (most cited first, 5 of 5)

- [Why Fine-Tuning Encourages Hallucinations and How to Fix It](https://mlsanity.com/paper/2604.15574/) (2026)
- [Harness Continual Learning: Continual Adaptation Beyond Model Parameters](https://mlsanity.com/paper/2608.19013/) (2026)
- [Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2602.20796/) (2026)
- [Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shift](https://mlsanity.com/paper/2605.27469/) (2026)
- [Geometry-Anchored Transport Framework for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2606.25347/) (2026)

## Abstract (arXiv)

In continual learning scenarios, catastrophic forgetting of previously learned tasks is a critical issue, making it essential to effectively measure such forgetting. Recently, there has been growing interest in focusing on representation forgetting, the forgetting measured at the hidden layer. In this paper, we provide the first theoretical analysis of representation forgetting and use this analysis to better understand the behavior of continual learning. First, we introduce a new metric called representation discrepancy, which measures the difference between representation spaces constructed by two snapshots of a model trained through continual learning. We demonstrate that our proposed metric serves as an effective surrogate for the representation forgetting while remaining analytically tractable. Second, through mathematical analysis of our metric, we derive several key findings about the dynamics of representation forgetting: the forgetting occurs more rapidly to a higher degree as the layer index increases, while increasing the width of the network slows down the forgetting process. Third, we support our theoretical findings through experiments on real image datasets, including Split-CIFAR100 and ImageNet1K.