mlsanityData to 7 Oct 2026

Continual Learning of Natural Language Processing Tasks: A Survey

arXiv 2211.12701 ↗First posted 23 Nov 2022cs.CL cs.AI cs.LGcited by 83

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) is a learning paradigm that emulates the human capability of learning and accumulating knowledge continually without forgetting the previously learned knowledge and also transferring the learned knowledge to help learn new tasks better. This survey presents a comprehensive review and analysis of the recent progress of CL in NLP, which has significant differences from CL in computer vision and machine learning. It covers (1) all CL settings with a taxonomy of existing techniques; (2) catastrophic forgetting (CF) prevention, (3) knowledge transfer (KT), which is particularly important for NLP tasks; and (4) some theory and the hidden challenge of inter-task class separation (ICS). (1), (3) and (4) have not been included in the existing survey. Finally, a list of future directions is discussed.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 79 of its 113 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

79 of 113 references.

References · 79

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2016Learning without Forgettingcited by 2,465
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2016Progressive Neural Networkscited by 1,332
38 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  2. 2017Continual Learning with Deep Generative Replaycited by 1,152
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2021Learning to Prompt for Continual Learningcited by 938
  5. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  6. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  9. 2017Dynamic Routing Between Capsulescited by 689
  10. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  11. 2019Three scenarios for continual learningcited by 610
  12. 2019Gradient based sample selection for online continual learningcited by 568
  13. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  14. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  15. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  16. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  17. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  18. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  19. 2019Continual learning with hypernetworkscited by 267
  20. 2020Few-Shot Class-Incremental Learningcited by 249
  21. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  22. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  23. 2020Supermasks in Superpositioncited by 223
  24. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  25. 2019Episodic Memory in Lifelong Language Learningcited by 188
  26. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  27. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  28. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  29. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  30. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  31. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  32. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  33. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  34. 2020A Comprehensive Study of Class Incremental Learning Algorithms for Visual Taskscited by 107
  35. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  36. 2019Uncertainty-guided Continual Learning with Bayesian Neural Networkscited by 100
  37. 2019Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challengescited by 92
  38. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 31; 2025: 20; 2026: 12.

83 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  4. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  5. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  6. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  7. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  8. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  9. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  10. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
  11. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  12. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
38 more
  1. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  2. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  3. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 37
  4. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  5. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
  6. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  7. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30
  8. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  9. 2024Rehearsal-Free Modular and Compositional Continual Learning for Language Modelscited by 26
  10. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 23
  11. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  12. 2023Investigating Forgetting in Pre-Trained Representations Through Continual Learningcited by 19
  13. 2024TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidationcited by 18
  14. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 17
  15. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  16. 2023GRASP: A Rehearsal Policy for Efficient Online Continual Learningcited by 13
  17. 2024Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attackscited by 13
  18. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  19. 2024SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Modelscited by 11
  20. 2023Continual Learning Under Language Shiftcited by 9
  21. 2024Class-incremental Learning for Time Series: Benchmark and Evaluationcited by 9
  22. 2024Continual Learning for Smart City: A Surveycited by 9
  23. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
  24. 2025CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generationcited by 9
  25. 2024O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editingcited by 8
  26. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  27. 2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
  28. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  29. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  30. 2025LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactionscited by 7
  31. 2025Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Modelscited by 7
  32. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7
  33. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 7
  34. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  35. 2024Structural Knowledge Informed Continual Multivariate Time Series Forecastingcited by 5
  36. 2024CLoG: Benchmarking Continual Learning of Image Generation Modelscited by 5
  37. 2024Boosting LLM Translation Skills without General Ability Loss via Rationale Distillationcited by 5
  38. 2024Continual Learning Using Only Large Language Model Promptingcited by 5

50 most cited of 83 All 83 (JSON)