mlsanityData to 7 Oct 2026

Is forgetting less a good inductive bias for forward transfer?

arXiv 2303.08207 ↗First posted 14 Mar 2023cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

One of the main motivations of studying continual learning is that the problem setting allows a model to accrue knowledge from past tasks to learn new tasks more efficiently. However, recent studies suggest that the key metric that continual learning algorithms optimize, reduction in catastrophic forgetting, does not correlate well with the forward transfer of knowledge. We believe that the conclusion previous works reached is due to the way they measure forward transfer. We argue that the measure of forward transfer to a task should not be affected by the restrictions placed on the continual learner in order to preserve knowledge of previous tasks. Instead, forward transfer should be measured by how easy it is to learn a new task given a set of representations produced by continual learning on previous tasks. Under this notion of forward transfer, we evaluate different continual learning algorithms on a variety of image classification benchmarks. Our results indicate that less forgetful representations lead to a better forward transfer suggesting a strong correlation between retaining past information and learning efficiency on new tasks. Further, we found less forgetful representations to be more diverse and discriminative compared to their forgetful counterparts.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

29 of 43 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  8. 2018Experience Replay for Continual Learningcited by 1,074
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  12. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
17 more
  1. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  2. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  3. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  4. 2017Variational Continual Learningcited by 335
  5. 2020Supermasks in Superpositioncited by 223
  6. 2019Meta-Learning Representations for Continual Learningcited by 189
  7. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  8. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  9. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  10. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  11. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  12. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  13. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 92
  14. 2020Learning to Continually Learncited by 82
  15. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  16. 2022Architecture Matters in Continual Learningcited by 46
  17. 2020Incremental Learning via Rate Reductioncited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 6; 2025: 5; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024LeanAgent: Lifelong Learning for Formal Theorem Provingcited by 34
  2. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  3. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  4. 2023Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Modelscited by 13
  5. 2024Hyperparameters in Continual Learning: A Reality Checkcited by 11
  6. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  7. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Promptingcited by 7
  8. 2025Sequence Transferability and Task Order Selection in Continual Learningcited by 5
  9. 2024Defying Imbalanced Forgetting in Class Incremental Learningcited by 4
  10. 2023Continual Learners are Incremental Model Generalizerscited by 3
  11. 2024Can LLMs Learn New Concepts Incrementally without Forgetting?cited by 1
  12. 2024Balancing the Causal Effects in Class-Incremental Learningcited by 1
6 more
  1. 2025CurLL: A Developmental Framework to Evaluate Continual Learning in Language Modelscited by 1
  2. 2023On consequences of finetuning on data with highly discriminative featurescited by 0
  3. 2025Revisiting Continual Semantic Segmentation with Pre-trained Vision Modelscited by 0
  4. 2025Empirical Evidences for the Effects of Feature Diversity in Open Set Recognition and Continual Learningcited by 0
  5. 2026Compensating Visual Insufficiency with Stratified Language Guidance for Long-Tail Class Incremental Learningcited by 0
  6. 2026In Two Minds about Lifelong Learning: Exploring Hemispheric Redundancy and Specialisation in Neural Modelscited by 0