mlsanityData to 7 Oct 2026

Progressive Prompts: Continual Learning for Language Models

arXiv 2301.12314 ↗First posted 29 Jan 2023cs.CL cs.AI cs.LGcited by 195

Not on the map. Why?

Abstract (arXiv)

We introduce Progressive Prompts - a simple and efficient approach for continual learning in language models. Our method allows forward transfer and resists catastrophic forgetting, without relying on data replay or a large number of task-specific parameters. Progressive Prompts learns a new soft prompt for each task and sequentially concatenates it with the previously learned prompts, while keeping the base model frozen. Experiments on standard continual learning benchmarks show that our approach outperforms state-of-the-art methods, with an improvement >20% in average test accuracy over the previous best-preforming method on T5 model. We also explore a more challenging continual learning setup with longer sequences of tasks and show that Progressive Prompts significantly outperforms prior methods.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 29 Jan 2023.

Builds on

30 of 40 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2016Learning without Forgettingcited by 2,465
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
18 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2018Experience Replay for Continual Learningcited by 1,074
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  6. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  8. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  9. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  10. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  11. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  12. 2019Episodic Memory in Lifelong Language Learningcited by 188
  13. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  14. 2019Continual Unsupervised Representation Learningcited by 147
  15. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  16. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  17. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  18. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 75; 2025: 53; 2026: 41.

195 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  5. 2023Mitigating the Alignment Tax of RLHFcited by 181
  6. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  7. 2024Continual Learning for Large Language Models: A Surveycited by 144
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  9. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  10. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  11. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  12. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
38 more
  1. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  2. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  3. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  4. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
  5. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  6. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  7. 2024Consistent Prompting for Rehearsal-Free Continual Learningcited by 52
  8. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  9. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 37
  10. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  11. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  12. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 30
  13. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 29
  14. 2023Learning to Modulate pre-trained Models in RLcited by 28
  15. 2024Scalable Language Model with Generalized Continual Learningcited by 28
  16. 2023Emptying the Ocean with a Spoon: Should We Edit Models?cited by 27
  17. 2023Continual Learning with Low Rank Adaptationcited by 27
  18. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  19. 2024Rehearsal-Free Modular and Compositional Continual Learning for Language Modelscited by 26
  20. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
  21. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 23
  22. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  23. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  24. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  25. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  26. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  27. 2023Multimodal Parameter-Efficient Few-Shot Class Incremental Learningcited by 19
  28. 2024HFT: Half Fine-Tuning for Large Language Modelscited by 18
  29. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
  30. 2024SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Modelscited by 17
  31. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16
  32. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  33. 2024Is Parameter Collision Hindering Continual Learning in LLMs?cited by 15
  34. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  35. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  36. 2024Empowering Large Language Model for Continual Video Question Answering with Collaborative Promptingcited by 14
  37. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  38. 2025Handling Out-of-Distribution Data: A Surveycited by 14

50 most cited of 195 All 195 (JSON)