mlsanityData to 14 Oct 2026

SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model

arXiv 2303.05118 ↗First posted 9 Mar 2023cs.CV cs.AI cs.LGcited by 180

Not on the map. Why?

Abstract (arXiv)

The goal of continual learning is to improve the performance of recognition models in learning sequentially arrived data. Although most existing works are established on the premise of learning from scratch, growing efforts have been devoted to incorporating the benefits of pre-training. However, how to adaptively exploit the pre-trained knowledge for each incremental task while maintaining its generalizability remains an open question. In this work, we present an extensive analysis for continual learning on a pre-trained model (CLPM), and attribute the key challenge to a progressive overfitting problem. Observing that selectively reducing the learning rate can almost resolve this issue in the representation layer, we propose a simple but extremely effective approach named Slow Learner with Classifier Alignment (SLCA), which further improves the classification layer by modeling the class-wise distributions and aligning the classification layers in a post-hoc fashion. Across a variety of scenarios, our proposal provides substantial improvements for CLPM (e.g., up to 49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively), and thus outperforms state-of-the-art approaches by a large margin. Based on such a strong baseline, critical factors and promising directions are analyzed in-depth to facilitate subsequent research. Code has been made available at: https://github.com/GengDavid/SLCA.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

43 of 66 references.

References · 43

  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2016Learning without Forgettingcited by 2,496
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  11. 2016Progressive Neural Networkscited by 1,361
  12. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
31 more
  1. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  2. 2021Learning to Prompt for Continual Learningcited by 944
  3. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  5. 2019Large Scale Incremental Learningcited by 732
  6. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  7. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  8. 2019Three scenarios for continual learningcited by 614
  9. 2021ImageNet-21K Pretraining for the Massescited by 613
  10. 2018Rethinking ImageNet Pre-trainingcited by 377
  11. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
  12. 2018Learning without Memorizingcited by 250
  13. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  14. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  15. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  16. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
  17. 2021Self-Supervised Models are Continual Learnerscited by 149
  18. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  19. 2021Few-Shot Segmentation via Cycle-Consistent Transformercited by 133
  20. 2019Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysiscited by 92
  21. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  22. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  23. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  24. 2022Memory Replay with Data Compression for Continual Learningcited by 82
  25. 2021AFEC: Active Forgetting of Negative Transfer in Continual Learningcited by 73
  26. 2021ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learningcited by 68
  27. 2022Class-Incremental Learning with Strong Pre-trained Modelscited by 60
  28. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  29. 2020Triple Memory Networks: a Brain-Inspired Method for Continual Learningcited by 28
  30. 2022Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentationcited by 28
  31. 2022Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanismcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 18; 2024: 66; 2025: 56; 2026: 40.

180 papers.

Sort:

By area:

  1. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  2. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 184
  3. 2023Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detectioncited by 182
  4. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 164
  5. 2024Continual Learning with Pre-Trained Models: A Surveycited by 154
  6. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  7. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 117
  8. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  9. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
  10. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  11. 2024Consistent Prompting for Rehearsal-Free Continual Learningcited by 54
  12. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
38 more
  1. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  2. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  3. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 42
  4. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  5. 2024Visual Prompt Tuning in Null Space for Continual Learningcited by 31
  6. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 31
  7. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservationcited by 29
  8. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  9. 2025Language Guided Concept Bottleneck Models for Interpretable Continual Learningcited by 28
  10. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 27
  11. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  12. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  13. 2024Closed-form merging of parameter-efficient modules for Federated Continual Learningcited by 22
  14. 2024OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learningcited by 20
  15. 2024HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuningcited by 20
  16. 2024PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learningcited by 19
  17. 2024SFC: Shared Feature Calibration in Weakly Supervised Semantic Segmentationcited by 19
  18. 2024Semantic Residual Prompts for Continual Learningcited by 19
  19. 2025Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learningcited by 19
  20. 2023CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentationcited by 18
  21. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  22. 2024SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Modelscited by 18
  23. 2025LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learningcited by 18
  24. 2023PromptFusion: Decoupling Stability and Plasticity for Continual Learningcited by 17
  25. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learningcited by 17
  26. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 17
  27. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  28. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  29. 2024Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learningcited by 16
  30. 2024Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentationcited by 15
  31. 2025Rethinking Query-based Transformer for Continual Image Segmentationcited by 14
  32. 2025Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learningcited by 14
  33. 2024MAGR: Manifold-Aligned Graph Regularization for Continual Action Quality Assessmentcited by 13
  34. 2024Dynamic Integration of Task-Specific Adapters for Class Incremental Learningcited by 13
  35. 2025Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Modelscited by 13
  36. 2025Bisecle: Binding and Separation in Continual Learning for Video Language Understandingcited by 13
  37. 2025Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Expertscited by 13
  38. 2024Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learningcited by 12

50 most cited of 180 All 180 (JSON)