# On the Stability-Plasticity Dilemma of Class-Incremental Learning

arXiv: https://arxiv.org/abs/2304.01663 · HTML: https://mlsanity.com/paper/2304.01663/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 4 Apr 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020)
- [SS-IL: Separated Softmax for Incremental Learning](https://mlsanity.com/paper/2003.13947/) (2020)
- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Drop to Adapt: Learning Discriminative Features for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1910.05562/) (2019)
- [Class-Incremental Learning for Action Recognition in Videos](https://mlsanity.com/paper/2203.13611/) (2022)
- [Energy-based Latent Aligner for Incremental Learning](https://mlsanity.com/paper/2203.14952/) (2022)
- [Formalizing the Generalization-Forgetting Trade-off in Continual Learning](https://mlsanity.com/paper/2109.14035/) (2021)
- [Continual Learning by Asymmetric Loss Approximation with Single-Side Overestimation](https://mlsanity.com/paper/1908.02984/) (2019)

## Built on by (most cited first, 30 of 56)

- [MagMax: Leveraging Model Merging for Seamless Continual Learning](https://mlsanity.com/paper/2407.06322/) (2024)
- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [Federated Continual Learning: Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2502.07059/) (2025)
- [Parameter-Efficient Continual Fine-Tuning: A Survey](https://mlsanity.com/paper/2504.13822/) (2025)
- [RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging](https://mlsanity.com/paper/2508.03121/) (2025)
- [Accelerating Heterogeneous Federated Learning with Closed-form Classifiers](https://mlsanity.com/paper/2406.01116/) (2024)
- [Revisiting Machine Unlearning with Dimensional Alignment](https://mlsanity.com/paper/2407.17710/) (2024)
- [FedProK: Trustworthy Federated Class-Incremental Learning via Prototypical Feature Knowledge Transfer](https://mlsanity.com/paper/2405.02685/) (2024)
- [BECAME: BayEsian Continual Learning with Adaptive Model MErging](https://mlsanity.com/paper/2504.02666/) (2025)
- [Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning](https://mlsanity.com/paper/2603.00191/) (2026)
- [Regularizing with Pseudo-Negatives for Continual Self-Supervised Learning](https://mlsanity.com/paper/2306.05101/) (2023)
- [PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2407.14029/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Continual Learning with Strong Experience Replay](https://mlsanity.com/paper/2305.13622/) (2023)
- [Early Preparation Pays Off: New Classifier Pre-tuning for Class Incremental Semantic Segmentation](https://mlsanity.com/paper/2407.14142/) (2024)
- [PLAN: Proactive Low-Rank Allocation for Continual Learning](https://mlsanity.com/paper/2510.21188/) (2025)
- [FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff](https://mlsanity.com/paper/2602.08040/) (2026)
- [Online Continual Learning via Logit Adjusted Softmax](https://mlsanity.com/paper/2311.06460/) (2023)
- [Revisiting Supervision for Continual Representation Learning](https://mlsanity.com/paper/2311.13321/) (2023)
- [RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detection](https://mlsanity.com/paper/2508.13878/) (2025)
- [Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning](https://mlsanity.com/paper/2403.18266/) (2024)
- [Random Registers for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2506.02843/) (2025)
- [SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models](https://mlsanity.com/paper/2602.07616/) (2026)
- [Look-Ahead Selective Plasticity for Continual Learning of Visual Tasks](https://mlsanity.com/paper/2311.01617/) (2023)
- [Mitigating the Stability-Plasticity Dilemma in Adaptive Train Scheduling with Curriculum-Driven Continual DQN Expansion](https://mlsanity.com/paper/2408.09838/) (2024)
- [A New Perspective to Boost Performance Fairness for Medical Federated Learning](https://mlsanity.com/paper/2410.19765/) (2024)
- [EXACFS -- A CIL Method to mitigate Catastrophic Forgetting](https://mlsanity.com/paper/2410.23751/) (2024)
- [Replay to Remember (R2R): An Efficient Uncertainty-driven Unsupervised Continual Learning Framework Using Generative Replay](https://mlsanity.com/paper/2505.04787/) (2025)

## Abstract (arXiv)

A primary goal of class-incremental learning is to strike a balance between stability and plasticity, where models should be both stable enough to retain knowledge learned from previously seen classes, and plastic enough to learn concepts from new classes. While previous works demonstrate strong performance on class-incremental benchmarks, it is not clear whether their success comes from the models being stable, plastic, or a mixture of both. This paper aims to shed light on how effectively recent class-incremental learning algorithms address the stability-plasticity trade-off. We establish analytical tools that measure the stability and plasticity of feature representations, and employ such tools to investigate models trained with various algorithms on large-scale class-incremental benchmarks. Surprisingly, we find that the majority of class-incremental learning algorithms heavily favor stability over plasticity, to the extent that the feature extractor of a model trained on the initial set of classes is no less effective than that of the final incremental model. Our observations not only inspire two simple algorithms that highlight the importance of feature representation analysis, but also suggest that class-incremental learning approaches, in general, should strive for better feature representation learning.