# Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need

arXiv: https://arxiv.org/abs/2303.07338 · HTML: https://mlsanity.com/paper/2303.07338/

## Key facts

- Cited by 184 papers in the map (data through 14 Oct 2026).
- Builds on 106 of its 166 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 30 of 106)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)

## Built on by (most cited first, 30 of 184)

- [RanPAC: Random Projections and Pre-trained Models for Continual Learning](https://mlsanity.com/paper/2307.02251/) (2023)
- [Continual Learning with Pre-Trained Models: A Survey](https://mlsanity.com/paper/2401.16386/) (2024)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Learning without Forgetting for Vision-Language Models](https://mlsanity.com/paper/2305.19270/) (2023)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [PILOT: A Pre-Trained Model-Based Continual Learning Toolbox](https://mlsanity.com/paper/2309.07117/) (2023)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion](https://mlsanity.com/paper/2407.14143/) (2024)
- [Mixture of Experts Meets Prompt-Based Continual Learning](https://mlsanity.com/paper/2405.14124/) (2024)
- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2407.10281/) (2024)
- [CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2505.24816/) (2025)
- [CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models](https://mlsanity.com/paper/2403.19137/) (2024)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Generative Multi-modal Models are Good Class-Incremental Learners](https://mlsanity.com/paper/2403.18383/) (2024)
- [OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learning](https://mlsanity.com/paper/2402.04129/) (2024)
- [Class Incremental Learning via Likelihood Ratio Based Task Prediction](https://mlsanity.com/paper/2309.15048/) (2023)
- [Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning](https://mlsanity.com/paper/2508.08165/) (2025)
- [SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models](https://mlsanity.com/paper/2411.02175/) (2024)
- [Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning](https://mlsanity.com/paper/2412.18219/) (2024)
- [LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2503.18985/) (2025)
- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning](https://mlsanity.com/paper/2410.00911/) (2024)
- [Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Prompt](https://mlsanity.com/paper/2403.09857/) (2024)
- [Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2509.16738/) (2025)
- [Dynamic Integration of Task-Specific Adapters for Class Incremental Learning](https://mlsanity.com/paper/2409.14983/) (2024)
- [Componential Prompt-Knowledge Alignment for Domain Incremental Learning](https://mlsanity.com/paper/2505.04575/) (2025)
- [Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Experts](https://mlsanity.com/paper/2507.07100/) (2025)
- [Hyperparameters in Continual Learning: A Reality Check](https://mlsanity.com/paper/2403.09066/) (2024)

## Abstract (arXiv)

Class-incremental learning (CIL) aims to adapt to emerging new classes without forgetting old ones. Traditional CIL models are trained from scratch to continually acquire knowledge as data evolves. Recently, pre-training has achieved substantial progress, making vast pre-trained models (PTMs) accessible for CIL. Contrary to traditional methods, PTMs possess generalizable embeddings, which can be easily transferred for CIL. In this work, we revisit CIL with PTMs and argue that the core factors in CIL are adaptivity for model updating and generalizability for knowledge transferring. 1) We first reveal that frozen PTM can already provide generalizable embeddings for CIL. Surprisingly, a simple baseline (SimpleCIL) which continually sets the classifiers of PTM to prototype features can beat state-of-the-art even without training on the downstream task. 2) Due to the distribution gap between pre-trained and downstream datasets, PTM can be further cultivated with adaptivity via model adaptation. We propose AdaPt and mERge (APER), which aggregates the embeddings of PTM and adapted models for classifier construction. APER is a general framework that can be orthogonally combined with any parameter-efficient tuning method, which holds the advantages of PTM's generalizability and adapted model's adaptivity. 3) Additionally, considering previous ImageNet-based benchmarks are unsuitable in the era of PTM due to data overlapping, we propose four new benchmarks for assessment, namely ImageNet-A, ObjectNet, OmniBenchmark, and VTAB. Extensive experiments validate the effectiveness of APER with a unified and concise framework. Code is available at https://github.com/zhoudw-zdw/RevisitingCIL