mlsanityData to 7 Oct 2026

LAMOL: LAnguage MOdeling for Lifelong Language Learning

arXiv 1909.03329 ↗First posted 7 Sep 2019cs.CL cs.AIcited by 156

Not on the map. Why?

Abstract (arXiv)

Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/jojotenya/LAMOL.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 28; 2023: 27; 2024: 37; 2025: 19; 2026: 19.

156 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2024LoRA Learns Less and Forgets Lesscited by 313
  3. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  4. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  5. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  6. 2023Mitigating the Alignment Tax of RLHFcited by 181
  7. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  8. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  9. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  10. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  11. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  12. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
38 more
  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  3. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  4. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  5. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  6. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  7. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  8. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  9. 2023On the Creativity of Large Language Modelscited by 57
  10. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  11. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  12. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  13. 2022Memory Efficient Continual Learning with Transformerscited by 43
  14. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  15. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  16. 2024A Unified Replay-based Continuous Learning Framework for Spatio-Temporal Prediction on Streaming Datacited by 39
  17. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  18. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 37
  19. 2021Distilled Replay: Overcoming Forgetting through Synthetic Samplescited by 36
  20. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35
  21. 2022Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Taskcited by 35
  22. 2021Curriculum-Meta Learning for Order-Robust Continual Relation Extractioncited by 34
  23. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  24. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  25. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfercited by 31
  26. 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
  27. 2022Consistent Representation Learning for Continual Relation Extractioncited by 28
  28. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  29. 2022ConTinTin: Continual Learning from Task Instructionscited by 26
  30. 2022Meta Learning for Natural Language Processing: A Surveycited by 26
  31. 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
  32. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  33. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  34. 2022On Continual Model Refinement in Out-of-Distribution Data Streamscited by 22
  35. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  36. 2021Towards Lifelong Learning of End-to-end ASRcited by 20
  37. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  38. 2021Prototype-Guided Memory Replay for Continual Learningcited by 19

50 most cited of 156 All 156 (JSON)