# An Efficient Replay for Class-Incremental Learning with Pre-trained Models

arXiv: https://arxiv.org/abs/2408.08084 · HTML: https://mlsanity.com/paper/2408.08084/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Builds on (most cited first, 22 of 22)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Progressive Prompts: Continual Learning for Language Models](https://mlsanity.com/paper/2301.12314/) (2023)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [Continual Learning with Pre-Trained Models: A Survey](https://mlsanity.com/paper/2401.16386/) (2024)
- [A Unified Continual Learning Framework with General Parameter-Efficient Tuning](https://mlsanity.com/paper/2303.10070/) (2023)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning](https://mlsanity.com/paper/2210.04428/) (2022)
- [Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference](https://mlsanity.com/paper/2211.15969/) (2022)
- [Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models](https://mlsanity.com/paper/2305.09144/) (2023)
- [Efficient Data Perturbation for Privacy Preserving and Accurate Data Stream Mining](https://mlsanity.com/paper/1806.06151/) (2018)

## Built on by (most cited first, 2 of 2)

- [Replay to Remember (R2R): An Efficient Uncertainty-driven Unsupervised Continual Learning Framework Using Generative Replay](https://mlsanity.com/paper/2505.04787/) (2025)
- [I Detect What I Don't Know: Incremental Anomaly Learning with Stochastic Weight Averaging-Gaussian for Oracle-Free Medical Imaging](https://mlsanity.com/paper/2511.03912/) (2025)

## Abstract (arXiv)

In general class-incremental learning, researchers typically use sample sets as a tool to avoid catastrophic forgetting during continuous learning. At the same time, researchers have also noted the differences between class-incremental learning and Oracle training and have attempted to make corrections. In recent years, researchers have begun to develop class-incremental learning algorithms utilizing pre-trained models, achieving significant results. This paper observes that in class-incremental learning, the steady state among the weight guided by each class center is disrupted, which is significantly correlated with catastrophic forgetting. Based on this, we propose a new method to overcoming forgetting . In some cases, by retaining only a single sample unit of each class in memory for replay and applying simple gradient constraints, very good results can be achieved. Experimental results indicate that under the condition of pre-trained models, our method can achieve competitive performance with very low computational cost and by simply using the cross-entropy loss.