# Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector

arXiv: https://arxiv.org/abs/2110.07905 · HTML: https://mlsanity.com/paper/2110.07905/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [Uncertainty-based Continual Learning with Adaptive Regularization](https://mlsanity.com/paper/1905.11614/) (2019)
- [Linear Mode Connectivity in Multitask and Continual Learning](https://mlsanity.com/paper/2010.04495/) (2020)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild](https://mlsanity.com/paper/1903.12648/) (2019)
- [Encoder Based Lifelong Learning](https://mlsanity.com/paper/1704.01920/) (2017)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning](https://mlsanity.com/paper/2009.01797/) (2020)

## Built on by (most cited first, 30 of 34)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning](https://mlsanity.com/paper/2303.09483/) (2023)
- [Learning Federated Visual Prompt in Null Space for MRI Reconstruction](https://mlsanity.com/paper/2303.16181/) (2023)
- [SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models](https://mlsanity.com/paper/2503.07392/) (2025)
- [Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning](https://mlsanity.com/paper/2311.17597/) (2023)
- [Neuroplastic Expansion in Deep Reinforcement Learning](https://mlsanity.com/paper/2410.07994/) (2024)
- [Task-Agnostic Guided Feature Expansion for Class-Incremental Learning](https://mlsanity.com/paper/2503.00823/) (2025)
- [Continual Gradient Low-Rank Projection Fine-Tuning for LLMs](https://mlsanity.com/paper/2507.02503/) (2025)
- [LW2G: Learning Whether to Grow for Prompt-based Continual Learning](https://mlsanity.com/paper/2409.18860/) (2024)
- [BECAME: BayEsian Continual Learning with Adaptive Model MErging](https://mlsanity.com/paper/2504.02666/) (2025)
- [Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning](https://mlsanity.com/paper/2603.00191/) (2026)
- [PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2407.14029/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning](https://mlsanity.com/paper/2505.22389/) (2025)
- [Adversarial Robust Memory-Based Continual Learner](https://mlsanity.com/paper/2311.17608/) (2023)
- [Flatness-aware Sequential Learning Generates Resilient Backdoors](https://mlsanity.com/paper/2407.14738/) (2024)
- [On the Effectiveness of Equivariant Regularization for Robust Online Continual Learning](https://mlsanity.com/paper/2305.03648/) (2023)
- [Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learning](https://mlsanity.com/paper/2406.08830/) (2024)
- [Flashbacks to Harmonize Stability and Plasticity in Continual Learning](https://mlsanity.com/paper/2506.00477/) (2025)
- [Continual Learning through Networks Splitting and Merging with Dreaming-Meta-Weighted Model Fusion](https://mlsanity.com/paper/2312.07082/) (2023)
- [Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learning](https://mlsanity.com/paper/2505.22358/) (2025)
- [Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective](https://mlsanity.com/paper/2602.02767/) (2026)
- [Fixed Random Classifier Rearrangement for Continual Learning](https://mlsanity.com/paper/2402.15227/) (2024)
- [The Championship-Winning Solution for the 5th CLVISION Challenge 2024](https://mlsanity.com/paper/2406.16615/) (2024)
- [The Solution for the sequential task continual learning track of the 2nd Greater Bay Area International Algorithm Competition](https://mlsanity.com/paper/2407.04996/) (2024)
- [One Stone, Four Birds: A Comprehensive Solution for QA System Using Supervised Contrastive Learning](https://mlsanity.com/paper/2407.09011/) (2024)
- [Task-conditioned Ensemble of Expert Models for Continuous Learning](https://mlsanity.com/paper/2504.08626/) (2025)
- [CL-BioGAN: Biologically-Inspired Cross-Domain Continual Learning for Hyperspectral Anomaly Detection](https://mlsanity.com/paper/2505.11796/) (2025)
- [Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions](https://mlsanity.com/paper/2509.19159/) (2025)

## Abstract (arXiv)

Plasticity-stability dilemma is a main problem for incremental learning, where plasticity is referring to the ability to learn new knowledge, and stability retains the knowledge of previous tasks. Many methods tackle this problem by storing previous samples, while in some applications, training data from previous tasks cannot be legally stored. In this work, we propose to employ mode connectivity in loss landscapes to achieve better plasticity-stability trade-off without any previous samples. We give an analysis of why and how to connect two independently optimized optima of networks, null-space projection for previous tasks and simple SGD for the current task, can attain a meaningful balance between preserving already learned knowledge and granting sufficient flexibility for learning a new task. This analysis of mode connectivity also provides us a new perspective and technology to control the trade-off between plasticity and stability. We evaluate the proposed method on several benchmark datasets. The results indicate our simple method can achieve notable improvement, and perform well on both the past and current tasks. On 10-split-CIFAR-100 task, our method achieves 79.79% accuracy, which is 6.02% higher. Our method also achieves 6.33% higher accuracy on TinyImageNet. Code is available at https://github.com/lingl1024/Connector.