mlsanityData to 7 Oct 2026

Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning

arXiv 2112.02706 ↗First posted 5 Dec 2021cs.CL cs.AI cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) learns a sequence of tasks incrementally with the goal of achieving two main objectives: overcoming catastrophic forgetting (CF) and encouraging knowledge transfer (KT) across tasks. However, most existing techniques focus only on overcoming CF and have no mechanism to encourage KT, and thus do not do well in KT. Although several papers have tried to deal with both CF and KT, our experiments show that they suffer from serious CF when the tasks do not have much shared knowledge. Another observation is that most current CL methods do not use pre-trained models, but it has been shown that such models can significantly improve the end task performance. For example, in natural language processing, fine-tuning a BERT-like pre-trained language model is one of the most effective approaches. However, for CL, this approach suffers from serious CF. An interesting question is how to make the best use of pre-trained models for CL. This paper proposes a novel model called CTR to solve these problems. Our experimental results demonstrate the effectiveness of CTR

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 5 Dec 2021.

Builds on

43 of 69 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2016Progressive Neural Networkscited by 1,332
  9. 2017Continual Learning with Deep Generative Replaycited by 1,152
  10. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  11. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  12. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
31 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  2. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  3. 2017Dynamic Routing Between Capsulescited by 689
  4. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  5. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  6. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  7. 2018Learning Word Vectors for 157 Languagescited by 279
  8. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  9. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  10. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  11. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  12. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  13. 2019Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentencecited by 121
  14. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  15. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  16. 2016Aspect Level Sentiment Classification with Deep Memory Networkcited by 82
  17. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  18. 2018Aspect Based Sentiment Analysis with Gated Convolutional Networkscited by 65
  19. 2017Deep Generative Dual Memory Network for Continual Learningcited by 59
  20. 2017Continual Learning in Generative Adversarial Netscited by 51
  21. 2019Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replaycited by 51
  22. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48
  23. 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
  24. 2020Lifelong Language Knowledge Distillationcited by 31
  25. 2020Neural Topic Modeling with Continual Lifelong Learningcited by 24
  26. 2019Continual Learning for Sentence Representations Using Conceptorscited by 22
  27. 2018Lifelong Learning for Sentiment Classificationcited by 21
  28. 2017Lifelong Learning CRF for Supervised Aspect Extractioncited by 9
  29. 2019Forward and Backward Knowledge Transfer for Sentiment Classificationcited by 9
  30. 2020Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learningcited by 9
  31. 2017Dual Attention Network for Product Compatibility and Function Satisfiability Analysiscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 25; 2024: 29; 2025: 16; 2026: 7.

91 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2023Continual Pre-training of Language Modelscited by 194
  4. 2022Can Foundation Models Wrangle Your Data?cited by 173
  5. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  6. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  7. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  8. 2023Continual Learning: Applications and the Road Forwardcited by 59
  9. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  10. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  11. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  12. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
38 more
  1. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  2. 2022Self-Supervised Continual Graph Learning in Adaptive Riemannian Spacescited by 32
  3. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
  4. 2023Learnability and Algorithm for Continual Learningcited by 28
  5. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  6. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  7. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
  8. 2024A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practicecited by 26
  9. 2022AI Autonomy : Self-Initiated Open-World Continual Learning and Adaptationcited by 23
  10. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  11. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  12. 2025Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generationcited by 23
  13. 2022Task-Free Continual Learning via Online Discrepancy Distance Learningcited by 22
  14. 2023Open-World Continual Learning: Unifying Novelty Detection and Continual Learningcited by 22
  15. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
  16. 2022Discrete Key-Value Bottleneckcited by 18
  17. 2024TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidationcited by 18
  18. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 17
  19. 2022A Theory for Knowledge Transfer in Continual Learningcited by 16
  20. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  21. 2024Continual Learning Using a Kernel-Based Method Over Foundation Modelscited by 13
  22. 2022Continuous Detection, Rapidly React: Unseen Rumors Detection based on Continual Prompt-Tuningcited by 12
  23. 2023Continual Task Allocation in Meta-Policy Network via Sparse Promptingcited by 11
  24. 2025Parameter-Efficient Continual Fine-Tuning: A Surveycited by 11
  25. 2023Lifelong Sequence Generation with Dynamic Module Expansion and Adaptationcited by 10
  26. 2024Boosting Large Language Models with Continual Learning for Aspect-based Sentiment Analysiscited by 10
  27. 2023Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extractioncited by 9
  28. 2024Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embeddingcited by 9
  29. 2024Q-Tuning: Queue-based Prompt Tuning for Lifelong Few-shot Language Learningcited by 9
  30. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
  31. 2022Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialoguecited by 8
  32. 2024Dynamic Transformer Architecture for Continual Learning of Multimodal Taskscited by 8
  33. 2023EvoluNet: Advancing Dynamic Non-IID Transfer Learning on Graphscited by 7
  34. 2023Continual Learning with Dirichlet Generative-based Rehearsalcited by 7
  35. 2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
  36. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  37. 2025Order-Robust Class Incremental Learning: Graph-Driven Dynamic Similarity Groupingcited by 7
  38. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7

50 most cited of 91 All 91 (JSON)