mlsanityData to 14 Oct 2026

Continual Learning with Pre-Trained Models: A Survey

arXiv 2401.16386 ↗First posted 29 Jan 2024cs.LG cs.CVcited by 154

Not on the map. Why?

Abstract (arXiv)

Nowadays, real-world applications often face streaming data, which requires the learning system to absorb new knowledge as data evolves. Continual Learning (CL) aims to achieve this goal and meanwhile overcome the catastrophic forgetting of former knowledge when learning new ones. Typical CL methods build the model from scratch to grow with incoming data. However, the advent of the pre-trained model (PTM) era has sparked immense research interest, particularly in leveraging PTMs' robust representational capabilities. This paper presents a comprehensive survey of the latest advancements in PTM-based CL. We categorize existing methodologies into three distinct groups, providing a comparative analysis of their similarities, differences, and respective advantages and disadvantages. Additionally, we offer an empirical study contrasting various state-of-the-art methods to highlight concerns regarding fairness in comparisons. The source code to reproduce these evaluations is available at: https://github.com/sun-hailong/LAMDA-PILOT

Key facts. Cited by 154 papers in the map (data through 14 Oct 2026). Builds on 54 of its 80 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

54 of 80 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2022Visual Prompt Tuningcited by 1,681
  7. 2019Natural Adversarial Examplescited by 1,408
  8. 2021Learning to Prompt for Continual Learningcited by 944
  9. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  10. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  11. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  12. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
38 more
  1. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  2. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  3. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  4. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  5. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  6. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
  7. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  8. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  9. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 184
  10. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
  11. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 164
  12. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  13. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 154
  14. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 125
  15. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  16. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  17. 2022Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptationcited by 98
  18. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  19. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  20. 2020MgSvF: Multi-Grained Slow vs. Fast Framework for Few-Shot Class-Incremental Learningcited by 62
  21. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  22. 2022ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protectioncited by 57
  23. 2022A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learningcited by 54
  24. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  25. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46
  26. 2023Few-Shot Class-Incremental Learning via Training-Free Prototype Calibrationcited by 46
  27. 2022PIVOT: Prompting for Video Continual Learningcited by 45
  28. 2022Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interferencecited by 44
  29. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 42
  30. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  31. 2022Benchmarking Omni-Vision Representation through the Lens of Visual Realmscited by 38
  32. 2023Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuningcited by 35
  33. 2022Self-Supervised Continual Graph Learning in Adaptive Riemannian Spacescited by 32
  34. 2022Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detectioncited by 23
  35. 2023Exploring Continual Learning for Code Generation Modelscited by 22
  36. 2023Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Modelscited by 18
  37. 2023PromptFusion: Decoupling Stability and Plasticity for Continual Learningcited by 17
  38. 2020Revisiting Meta-Learning as Supervised Learningcited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 38; 2025: 61; 2026: 55.

154 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 117
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  5. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  6. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  7. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  8. 2024MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learningcited by 36
  9. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 31
  10. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 28
  11. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  12. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
38 more
  1. 2024SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Modelscited by 18
  2. 2024Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learningcited by 18
  3. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 15
  4. 2025Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Modelscited by 14
  5. 2024Dynamic Integration of Task-Specific Adapters for Class Incremental Learningcited by 13
  6. 2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
  7. 2024Hyperparameters in Continual Learning: A Reality Checkcited by 12
  8. 2024Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Modelscited by 11
  9. 2025Parameter-Efficient Continual Fine-Tuning: A Surveycited by 11
  10. 2024Continual Adapter Tuning with Semantic Shift Compensation for Class-Incremental Learningcited by 10
  11. 2025SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splittingcited by 10
  12. 2024Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformerscited by 9
  13. 2024LW2G: Learning Whether to Grow for Prompt-based Continual Learningcited by 9
  14. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  15. 2025Continuous Subspace Optimization for Continual Learningcited by 9
  16. 2024Adaptive Adapter Routing for Long-Tailed Class-Incremental Learningcited by 8
  17. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 8
  18. 2025Navigating Semantic Drift in Task-Agnostic Class-Incremental Learningcited by 8
  19. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  20. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 8
  21. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuningcited by 8
  22. 2025Order-Robust Class Incremental Learning: Graph-Driven Dynamic Similarity Groupingcited by 7
  23. 2025IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Promptingcited by 7
  24. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  25. 2024Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learningcited by 6
  26. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projectioncited by 6
  27. 2024Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learningcited by 5
  28. 2024How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation modelscited by 5
  29. 2025RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detectioncited by 5
  30. 2026Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptationcited by 5
  31. 2024Future-Proofing Class-Incremental Learningcited by 4
  32. 2025Achieving Upper Bound Accuracy of Joint Training in Continual Learningcited by 4
  33. 2025Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Themcited by 4
  34. 2024Masked Autoencoders are Parameter-Efficient Federated Continual Learnerscited by 3
  35. 2025PrePrompt: Predictive prompting for class incremental learningcited by 3
  36. 2025Towards Efficient and Exact Forgetting Services in Pre-Trained-Model-based Continual Learningcited by 3
  37. 2025AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Datacited by 3
  38. 2025Policy Compatible Skill Incremental Learning via Lazy Learning Interfacecited by 3

50 most cited of 154 All 154 (JSON)