# Balancing Stability and Plasticity through Advanced Null Space in Continual Learning

arXiv: https://arxiv.org/abs/2207.12061 · HTML: https://mlsanity.com/paper/2207.12061/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 59 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 30 of 43)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [Measuring Catastrophic Forgetting in Neural Networks](https://mlsanity.com/paper/1708.02072/) (2017)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [Continual Learning of Context-dependent Processing in Neural Networks](https://mlsanity.com/paper/1810.01256/) (2018)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [Scalable and Order-robust Continual Learning with Additive Parameter Decomposition](https://mlsanity.com/paper/1902.09432/) (2019)

## Built on by (most cited first, 30 of 30)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Dense Network Expansion for Class Incremental Learning](https://mlsanity.com/paper/2303.12696/) (2023)
- [Learning Federated Visual Prompt in Null Space for MRI Reconstruction](https://mlsanity.com/paper/2303.16181/) (2023)
- [SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models](https://mlsanity.com/paper/2503.07392/) (2025)
- [The Ideal Continual Learner: An Agent That Never Forgets](https://mlsanity.com/paper/2305.00316/) (2023)
- [Visual Prompt Tuning in Null Space for Continual Learning](https://mlsanity.com/paper/2406.05658/) (2024)
- [Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation](https://mlsanity.com/paper/2310.15746/) (2023)
- [LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2503.18985/) (2025)
- [Exploring Data Geometry for Continual Learning](https://mlsanity.com/paper/2304.03931/) (2023)
- [Vector Quantization Prompting for Continual Learning](https://mlsanity.com/paper/2410.20444/) (2024)
- [Continual Gradient Low-Rank Projection Fine-Tuning for LLMs](https://mlsanity.com/paper/2507.02503/) (2025)
- [PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2407.14029/) (2024)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Restricted Orthogonal Gradient Projection for Continual Learning](https://mlsanity.com/paper/2301.12131/) (2023)
- [ScrollNet: Dynamic Weight Importance for Continual Learning](https://mlsanity.com/paper/2308.16567/) (2023)
- [Taxonomic Class Incremental Learning](https://mlsanity.com/paper/2304.05547/) (2023)
- [Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectives](https://mlsanity.com/paper/2307.02140/) (2023)
- [Continual Learning of Range-Dependent Transmission Loss for Underwater Acoustic using Conditional Convolutional Neural Net](https://mlsanity.com/paper/2404.08091/) (2024)
- [Replay-Free Continual Low-Rank Adaptation with Dynamic Memory](https://mlsanity.com/paper/2411.00623/) (2024)
- [Divide-and-Conquer for Enhancing Unlabeled Learning, Stability, and Plasticity in Semi-supervised Continual Learning](https://mlsanity.com/paper/2508.05316/) (2025)
- [DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces](https://mlsanity.com/paper/2509.19230/) (2025)
- [Keep Moving: identifying task-relevant subspaces to maximise plasticity for newly learned tasks](https://mlsanity.com/paper/2310.04741/) (2023)
- [Towards Plastic and Stable Exemplar-Free Incremental Learning: A Dual-Learner Framework with Cumulative Parameter Averaging](https://mlsanity.com/paper/2310.18639/) (2023)
- [InTAct: Interval-based Task Activation Consolidation for Continual Learning](https://mlsanity.com/paper/2511.17439/) (2025)
- [Continual Learning through Networks Splitting and Merging with Dreaming-Meta-Weighted Model Fusion](https://mlsanity.com/paper/2312.07082/) (2023)
- [Learning a Low-Rank Feature Representation: Achieving Better Trade-Off between Stability and Plasticity in Continual Learning](https://mlsanity.com/paper/2312.08740/) (2023)
- [CL-BioGAN: Biologically-Inspired Cross-Domain Continual Learning for Hyperspectral Anomaly Detection](https://mlsanity.com/paper/2505.11796/) (2025)
- [Diagnosing Shortcut-Induced Rigidity in Continual Learning: The Einstellung Rigidity Index (ERI)](https://mlsanity.com/paper/2510.00475/) (2025)
- [Attention Retention for Continual Learning with Vision Transformers](https://mlsanity.com/paper/2602.05454/) (2026)
- [A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning](https://mlsanity.com/paper/2610.04670/) (2026)

## Abstract (arXiv)

Continual learning is a learning paradigm that learns tasks sequentially with resources constraints, in which the key challenge is stability-plasticity dilemma, i.e., it is uneasy to simultaneously have the stability to prevent catastrophic forgetting of old tasks and the plasticity to learn new tasks well. In this paper, we propose a new continual learning approach, Advanced Null Space (AdNS), to balance the stability and plasticity without storing any old data of previous tasks. Specifically, to obtain better stability, AdNS makes use of low-rank approximation to obtain a novel null space and projects the gradient onto the null space to prevent the interference on the past tasks. To control the generation of the null space, we introduce a non-uniform constraint strength to further reduce forgetting. Furthermore, we present a simple but effective method, intra-task distillation, to improve the performance of the current task. Finally, we theoretically find that null space plays a key role in plasticity and stability, respectively. Experimental results show that the proposed method can achieve better performance compared to state-of-the-art continual learning approaches.