mlsanityData to 7 Oct 2026

Continual Lifelong Learning in Natural Language Processing: A Survey

arXiv 2012.09823 ↗First posted 17 Dec 2020cs.CL cs.AI cs.LGcited by 152

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) aims to enable information systems to learn from a continuous data stream across time. However, it is difficult for existing deep learning architectures to learn a new task without largely forgetting previously acquired knowledge. Furthermore, CL is particularly challenging for language learning, as natural language is ambiguous: it is discrete, compositional, and its meaning is context-dependent. In this work, we look at the problem of CL through the lens of various NLP tasks. Our survey discusses major challenges in CL and current methods applied in neural network models. We also provide a critical review of the existing CL evaluation methods and datasets in NLP. Finally, we present our outlook on future research directions.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 32; 2023: 40; 2024: 43; 2025: 18; 2026: 7.

152 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  4. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  5. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  6. 2023Class-Incremental Learning: A Surveycited by 209
  7. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  8. 2024Continual Learning for Large Language Models: A Surveycited by 144
  9. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  10. 2022Continual Learning and Private Unlearningcited by 124
  11. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 114
  12. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
38 more
  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  3. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  4. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  5. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  6. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  7. 2021Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworkscited by 56
  8. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  9. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  10. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  11. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  12. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  13. 2022Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Taskcited by 35
  14. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  15. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  16. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
  17. 2022Graph Lifelong Learning: A Surveycited by 30
  18. 2022ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examplescited by 29
  19. 2021CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparabilitycited by 28
  20. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  21. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  22. 2022ConTinTin: Continual Learning from Task Instructionscited by 26
  23. 2021RuleBert: Teaching Soft Rules to Pre-trained Language Modelscited by 25
  24. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  25. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  26. 2022Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularizationcited by 23
  27. 2022Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detectioncited by 23
  28. 2022On Continual Model Refinement in Out-of-Distribution Data Streamscited by 22
  29. 2023Exploring Continual Learning for Code Generation Modelscited by 22
  30. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  31. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  32. 2023Continual Learning of Diffusion Models with Generative Distillationcited by 19
  33. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  34. 2024Towards Robust Graph Incremental Learning on Evolving Graphscited by 18
  35. 2023Continual Graph Learning: A Surveycited by 17
  36. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 17
  37. 2024Graph Learning under Distribution Shifts: A Comprehensive Survey on Domain Adaptation, Out-of-distribution, and Continual Learningcited by 17
  38. 2025LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Trainingcited by 17

50 most cited of 152 All 152 (JSON)