mlsanityData to 14 Oct 2026

Transfer without Forgetting

arXiv 2206.00388 ↗First posted 1 Jun 2022cs.LG stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

This work investigates the entanglement between Continual Learning (CL) and Transfer Learning (TL). In particular, we shed light on the widespread application of network pretraining, highlighting that it is itself subject to catastrophic forgetting. Unfortunately, this issue leads to the under-exploitation of knowledge transfer during later tasks. On this ground, we propose Transfer without Forgetting (TwF), a hybrid approach building upon a fixed pretrained sibling network, which continuously propagates the knowledge inherent in the source domain through a layer-wise loss term. Our experiments indicate that TwF steadily outperforms other CL methods across a variety of settings, averaging a 4.81% gain in Class-Incremental accuracy over a variety of datasets and different buffer sizes.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 51 of its 73 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

51 of 73 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2017Mask R-CNNcited by 7,277
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  11. 2016Learning without Forgettingcited by 2,496
  12. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
38 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2016Progressive Neural Networkscited by 1,361
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  6. 2014How transferable are features in deep neural networks?cited by 1,225
  7. 2018Attention-based Deep Multiple Instance Learningcited by 1,095
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  9. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  10. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  11. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  12. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  13. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  14. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 612
  15. 2019Gradient based sample selection for online continual learningcited by 571
  16. 2018Born Again Neural Networkscited by 546
  17. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  18. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  19. 2020What is being transferred in transfer learning?cited by 368
  20. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  21. 2021Co2L: Contrastive Continual Learningcited by 244
  22. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  23. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 219
  24. 2018BAM: Bottleneck Attention Modulecited by 172
  25. 2017Modulating early visual processing by languagecited by 167
  26. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 161
  27. 2018Towards Robust Evaluations of Continual Learningcited by 148
  28. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  29. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  30. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  31. 2020Rethinking Experience Replay: a Bag of Tricks for Continual Learningcited by 94
  32. 2019Knowledge Distillation from Internal Representationscited by 87
  33. 2021Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworkscited by 56
  34. 2019Learning What and Where to Transfercited by 43
  35. 2022How many Observations are Enough? Knowledge Distillation for Trajectory Forecastingcited by 35
  36. 2021Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffercited by 30
  37. 2021Continual Semi-Supervised Learning through Contrastive Interpolation Consistencycited by 25
  38. 2020Subclass Distillationcited by 23

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 17; 2025: 10; 2026: 3.

44 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  3. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 63
  4. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  5. 2023Tangent Model Composition for Ensembling and Continual Fine-tuningcited by 25
  6. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  7. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 22
  8. 2023Adapter Learning in Pretrained Feature Extractor for Continual Learning of Diseasescited by 18
  9. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learningcited by 17
  10. 2023Continual Learning for Image Segmentation with Dynamic Querycited by 12
  11. 2023Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discoverycited by 11
  12. 2025SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splittingcited by 10
32 more
  1. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 8
  2. 2023Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoffcited by 7
  3. 2025StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learningcited by 7
  4. 2025Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learningcited by 7
  5. 2023Latent Spectral Regularization for Continual Learningcited by 6
  6. 2024DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical Imagescited by 6
  7. 2024Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retentioncited by 6
  8. 2025EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learningcited by 6
  9. 2025CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learningcited by 6
  10. 2024Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learningcited by 5
  11. 2024Background Adaptation with Residual Modeling for Exemplar-Free Class-Incremental Semantic Segmentationcited by 5
  12. 2022Schedule-Robust Online Continual Learningcited by 4
  13. 2023Exploring Robust Features for Improving Adversarial Robustnesscited by 4
  14. 2022Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learningcited by 3
  15. 2023Bayesian Transfer Learningcited by 3
  16. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  17. 2025C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Modelscited by 3
  18. 2024Improving Continual Learning Performance and Efficiency with Auxiliary Classifierscited by 2
  19. 2024Robust and Explainable Fine-Grained Visual Classification with Transfer Learning: A Dual-Carriageway Frameworkcited by 2
  20. 2024May the Forgetting Be with You: Alternate Replay for Learning with Noisy Labelscited by 2
  21. 2025One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learningcited by 2
  22. 2025MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Imagescited by 2
  23. 2023A baseline on continual learning methods for video action recognitioncited by 1
  24. 2024Selective Attention-based Modulation for Continual Learningcited by 1
  25. 2024FedRewind: Rewinding Continual Model Exchange for Decentralized Federated Learningcited by 1
  26. 2024Differentiable Prompt Learning for Vision Language Modelscited by 1
  27. 2024Lifelong Learning Using a Dynamically Growing Tree of Sub-networks for Domain Generalization in Video Object Segmentationcited by 0
  28. 2024An Attention-based Representation Distillation Baseline for Multi-Label Continual Learningcited by 0
  29. 2025Pathological Prior-Guided Multiple Instance Learning For Mitigating Catastrophic Forgetting in Breast Cancer Whole Slide Image Classificationcited by 0
  30. 2026Multi-Level Feature Fusion for Continual Learning in Visual Quality Inspectioncited by 0
  31. 2026Dream2Learn: Structured Generative Dreaming for Continual Learningcited by 0
  32. 2026Continual Model Merging with Test-Time Adaptation for Whole-Slide Image Analysiscited by 0