mlsanityData to 14 Oct 2026

Fine-tuned Language Models are Continual Learners

arXiv 2205.12393 ↗First posted 24 May 2022cs.CLcited by 156

Not on the map. Why?

Abstract (arXiv)

Recent work on large language models relies on the intuition that most natural language processing tasks can be described via natural language instructions. Language models trained on these instructions show strong zero-shot performance on several standard datasets. However, these models even though impressive still perform poorly on a wide range of tasks outside of their respective training and evaluation sets. To address this limitation, we argue that a model should be able to keep extending its knowledge and abilities, without forgetting previous skills. In spite of the limited success of Continual Learning we show that Language Models can be continual learners. We empirically investigate the reason for this success and conclude that Continual Learning emerges from self-supervision pre-training. Our resulting model Continual-T0 (CT0) is able to learn diverse new tasks, while still maintaining good performance on previous tasks, spanning remarkably through 70 datasets in total. Finally, we show that CT0 is able to combine instructions in ways it was never trained for, demonstrating some compositionality.

Key facts. Cited by 156 papers in the map (data through 14 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

32 of 49 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2017Continual Learning with Deep Generative Replaycited by 1,158
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  12. 2019ELI5: Long Form Question Answeringcited by 575
20 more
  1. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  2. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  3. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  4. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 289
  5. 2019Episodic Memory in Lifelong Language Learningcited by 188
  6. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  7. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  8. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  9. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  10. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  11. 2020ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformationscited by 101
  12. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 83
  13. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  14. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 67
  15. 2020Continual Learning for Natural Language Generation in Task-oriented Dialog Systemscited by 47
  16. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  17. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  18. 2022ConTinTin: Continual Learning from Task Instructionscited by 26
  19. 2022On Continual Model Refinement in Out-of-Distribution Data Streamscited by 22
  20. 2019Ask to Learn: A Study on Curiosity-driven Question Generationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 31; 2024: 55; 2025: 34; 2026: 34.

156 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2023A Comprehensive Overview of Large Language Modelscited by 648
  3. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  4. 2024LoRA Learns Less and Forgets Lesscited by 328
  5. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  7. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  8. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  9. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  10. 2024Continual Learning for Large Language Models: A Surveycited by 147
  11. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  12. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
38 more
  1. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  2. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  3. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
  4. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  5. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  6. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  7. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 99
  8. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  9. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  10. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  11. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  12. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  13. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 70
  14. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  15. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 66
  16. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  17. 2025End-to-End Test-Time Training for Long Contextcited by 62
  18. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 58
  19. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 57
  20. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 55
  21. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  22. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  23. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  24. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 40
  25. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  26. 2023Audio-Visual Class-Incremental Learningcited by 33
  27. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  28. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 33
  29. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  30. 2024Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptationcited by 27
  31. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 25
  32. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  33. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 24
  34. 2023LEMON: Lossless model expansioncited by 23
  35. 2024Unlocking Continual Learning Abilities in Language Modelscited by 23
  36. 2024Unleashing the Power of Continual Learning on Non-Centralized Devices: A Surveycited by 23
  37. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  38. 2024LLMs in the Imaginarium: Tool Learning through Simulated Trial and Errorcited by 19

50 most cited of 156 All 156 (JSON)