mlsanityData to 7 Oct 2026

Continual Training of Language Models for Few-Shot Learning

arXiv 2210.05549 ↗First posted 11 Oct 2022cs.CL cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. Adapting or posttraining an LM using an unlabeled domain corpus can produce even better performance for end-tasks in the domain. This paper proposes the problem of continually extending an LM by incrementally post-train the LM with a sequence of unlabeled domain corpora to expand its knowledge without forgetting its previous skills. The goal is to improve the few-shot end-task learning in these domains. The resulting system is called CPT (Continual PostTraining), which to our knowledge, is the first continual post-training system. Experimental results verify its effectiveness.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

48 of 64 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
36 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2017Continual Learning with Deep Generative Replaycited by 1,152
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  7. 2019Three scenarios for continual learningcited by 610
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  9. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  10. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  11. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  12. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  13. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  14. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  15. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  16. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  17. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  18. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  19. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  20. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  21. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  22. 2017Continual Learning in Generative Adversarial Netscited by 51
  23. 2017Few-Shot Learning Through an Information Retrieval Lenscited by 51
  24. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48
  25. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  26. 2018One-Shot Unsupervised Cross Domain Translationcited by 35
  27. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35
  28. 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
  29. 2020Lifelong Language Knowledge Distillationcited by 31
  30. 2021Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and Systemcited by 30
  31. 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
  32. 2022ConTinTin: Continual Learning from Task Instructionscited by 26
  33. 2019Continual Learning for Sentence Representations Using Conceptorscited by 22
  34. 2019Neural Snowball for Few-Shot Relation Learningcited by 22
  35. 2020Defining Benchmarks for Continual Few-Shot Learningcited by 21
  36. 2020Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 10; 2025: 4; 2026: 3.

33 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  4. 2023Continual Pre-training of Language Modelscited by 194
  5. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  6. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  7. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 77
  8. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  9. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  10. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  11. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  12. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
21 more
  1. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  2. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  3. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 17
  4. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
  5. 2023Class-Incremental Learning based on Label Generationcited by 11
  6. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglementcited by 11
  7. 2024Continual Learning for Smart City: A Surveycited by 9
  8. 2023FLatS: Principled Out-of-Distribution Detection with Feature-Based Likelihood Ratio Scorecited by 5
  9. 2024CLoG: Benchmarking Continual Learning of Image Generation Modelscited by 5
  10. 2023Primal Dual Continual Learning: Balancing Stability and Plasticity through Adaptive Memory Allocationcited by 4
  11. 2023Improving Few-shot Generalization of Safety Classifiers via Data Augmented Parameter-Efficient Fine-Tuningcited by 4
  12. 2023Continuous Training and Fine-tuning for Domain-Specific Language Models in Medical Question Answeringcited by 4
  13. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  14. 2024CEL: A Continual Learning Model for Disease Outbreak Prediction by Leveraging Domain Adaptation via Elastic Weight Consolidationcited by 3
  15. 2025AggTruth: Contextual Hallucination Detection using Aggregated Attention Scores in LLMscited by 2
  16. 2025Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generationcited by 1
  17. 2025T2I-ConBench: Text-to-Image Benchmark for Continual Post-trainingcited by 1
  18. 2025Tackling Distribution Shift in LLM via KILO: Knowledge-Instructed Learning for Continual Adaptationcited by 1
  19. 2024Continual Learning of Numerous Tasks from Long-tail Distributionscited by 0
  20. 2026ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AIcited by 0
  21. 2026Task-Oriented Rank Adaptation for Continual Learning in Text Classificationcited by 0