# Is forgetting less a good inductive bias for forward transfer?

arXiv: https://arxiv.org/abs/2303.08207 · HTML: https://mlsanity.com/paper/2303.08207/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [Linear Mode Connectivity in Multitask and Continual Learning](https://mlsanity.com/paper/2010.04495/) (2020)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Memory Efficient Experience Replay for Streaming Learning](https://mlsanity.com/paper/1809.05922/) (2018)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020)
- [Wide Neural Networks Forget Less Catastrophically](https://mlsanity.com/paper/2110.11526/) (2021)
- [Architecture Matters in Continual Learning](https://mlsanity.com/paper/2202.00275/) (2022)
- [Incremental Learning via Rate Reduction](https://mlsanity.com/paper/2011.14593/) (2020)

## Built on by (most cited first, 18 of 18)

- [LeanAgent: Lifelong Learning for Formal Theorem Proving](https://mlsanity.com/paper/2410.06209/) (2024)
- [Continual Learning in Linear Classification on Separable Data](https://mlsanity.com/paper/2306.03534/) (2023)
- [Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting](https://mlsanity.com/paper/2304.00933/) (2023)
- [Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models](https://mlsanity.com/paper/2312.07887/) (2023)
- [Hyperparameters in Continual Learning: A Reality Check](https://mlsanity.com/paper/2403.09066/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting](https://mlsanity.com/paper/2503.20612/) (2025)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [Defying Imbalanced Forgetting in Class Incremental Learning](https://mlsanity.com/paper/2403.14910/) (2024)
- [Continual Learners are Incremental Model Generalizers](https://mlsanity.com/paper/2306.12026/) (2023)
- [Can LLMs Learn New Concepts Incrementally without Forgetting?](https://mlsanity.com/paper/2402.08526/) (2024)
- [Balancing the Causal Effects in Class-Incremental Learning](https://mlsanity.com/paper/2402.10063/) (2024)
- [CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models](https://mlsanity.com/paper/2510.13008/) (2025)
- [On consequences of finetuning on data with highly discriminative features](https://mlsanity.com/paper/2310.19537/) (2023)
- [Revisiting Continual Semantic Segmentation with Pre-trained Vision Models](https://mlsanity.com/paper/2508.04267/) (2025)
- [Empirical Evidences for the Effects of Feature Diversity in Open Set Recognition and Continual Learning](https://mlsanity.com/paper/2508.13005/) (2025)
- [Compensating Visual Insufficiency with Stratified Language Guidance for Long-Tail Class Incremental Learning](https://mlsanity.com/paper/2603.21708/) (2026)
- [In Two Minds about Lifelong Learning: Exploring Hemispheric Redundancy and Specialisation in Neural Models](https://mlsanity.com/paper/2608.19514/) (2026)

## Abstract (arXiv)

One of the main motivations of studying continual learning is that the problem setting allows a model to accrue knowledge from past tasks to learn new tasks more efficiently. However, recent studies suggest that the key metric that continual learning algorithms optimize, reduction in catastrophic forgetting, does not correlate well with the forward transfer of knowledge. We believe that the conclusion previous works reached is due to the way they measure forward transfer. We argue that the measure of forward transfer to a task should not be affected by the restrictions placed on the continual learner in order to preserve knowledge of previous tasks. Instead, forward transfer should be measured by how easy it is to learn a new task given a set of representations produced by continual learning on previous tasks. Under this notion of forward transfer, we evaluate different continual learning algorithms on a variety of image classification benchmarks. Our results indicate that less forgetful representations lead to a better forward transfer suggesting a strong correlation between retaining past information and learning efficiency on new tasks. Further, we found less forgetful representations to be more diverse and discriminative compared to their forgetful counterparts.