# Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning

arXiv: https://arxiv.org/abs/2303.09483 · HTML: https://mlsanity.com/paper/2303.09483/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020)
- [Linear Mode Connectivity in Multitask and Continual Learning](https://mlsanity.com/paper/2010.04495/) (2020)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)
- [Conditional Channel Gated Networks for Task-Aware Continual Learning](https://mlsanity.com/paper/2004.00070/) (2020)
- [Less-forgetting Learning in Deep Neural Networks](https://mlsanity.com/paper/1607.00122/) (2016)

## Built on by (most cited first, 30 of 41)

- [Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning](https://mlsanity.com/paper/2402.18865/) (2024)
- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning](https://mlsanity.com/paper/2308.09544/) (2023)
- [Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective](https://mlsanity.com/paper/2506.03951/) (2025)
- [ProtoDepth: Unsupervised Continual Depth Completion with Prototypes](https://mlsanity.com/paper/2503.12745/) (2025)
- [BrainWash: A Poisoning Attack to Forget in Continual Learning](https://mlsanity.com/paper/2311.11995/) (2023)
- [Statistical Context Detection for Deep Lifelong Reinforcement Learning](https://mlsanity.com/paper/2405.19047/) (2024)
- [Continual Audio-Visual Sound Separation](https://mlsanity.com/paper/2411.02860/) (2024)
- [Why Fine-Tuning Encourages Hallucinations and How to Fix It](https://mlsanity.com/paper/2604.15574/) (2026)
- [Vision-Language Navigation with Continual Learning](https://mlsanity.com/paper/2409.02561/) (2024)
- [Modality-Inconsistent Continual Learning of Multimodal Large Language Models](https://mlsanity.com/paper/2412.13050/) (2024)
- [IBCL: Zero-shot Model Generation under Stability-Plasticity Trade-offs](https://mlsanity.com/paper/2305.14782/) (2023)
- [SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learning](https://mlsanity.com/paper/2405.09394/) (2024)
- [UnCLe: Benchmarking Unsupervised Continual Learning for Depth Completion](https://mlsanity.com/paper/2410.18074/) (2024)
- [Active Learning for Continual Learning: Keeping the Past Alive in the Present](https://mlsanity.com/paper/2501.14278/) (2025)
- [HeaRT: A Hierarchical Circuit Reasoning Tree-Based Agentic Framework for AMS Design Optimization](https://mlsanity.com/paper/2511.19669/) (2025)
- [Step Out and Seek Around: On Warm-Start Training with Incremental Data](https://mlsanity.com/paper/2406.04484/) (2024)
- [Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay](https://mlsanity.com/paper/2404.01828/) (2024)
- [Characterizing Continual Learning Scenarios and Strategies for Audio Analysis](https://mlsanity.com/paper/2407.00465/) (2024)
- [Divide-and-Conquer for Enhancing Unlabeled Learning, Stability, and Plasticity in Semi-supervised Continual Learning](https://mlsanity.com/paper/2508.05316/) (2025)
- [Elastic Weight Consolidation Done Right for Continual Learning](https://mlsanity.com/paper/2603.18596/) (2026)
- [Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers](https://mlsanity.com/paper/2403.07404/) (2024)
- [Continual Learning for Remote Physiological Measurement: Minimize Forgetting and Simplify Inference](https://mlsanity.com/paper/2407.13974/) (2024)
- [An Effective Dynamic Gradient Calibration Method for Continual Learning](https://mlsanity.com/paper/2407.20956/) (2024)
- [Mitigating the Stability-Plasticity Dilemma in Adaptive Train Scheduling with Curriculum-Driven Continual DQN Expansion](https://mlsanity.com/paper/2408.09838/) (2024)
- [Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticity](https://mlsanity.com/paper/2504.20932/) (2025)
- [Flashbacks to Harmonize Stability and Plasticity in Continual Learning](https://mlsanity.com/paper/2506.00477/) (2025)
- [InTAct: Interval-based Task Activation Consolidation for Continual Learning](https://mlsanity.com/paper/2511.17439/) (2025)
- [The Bayesian Approach to Continual Learning: An Overview](https://mlsanity.com/paper/2507.08922/) (2025)
- [Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning](https://mlsanity.com/paper/2509.12958/) (2025)

## Abstract (arXiv)

In contrast to the natural capabilities of humans to learn new tasks in a sequential fashion, neural networks are known to suffer from catastrophic forgetting, where the model's performances on old tasks drop dramatically after being optimized for a new task. Since then, the continual learning (CL) community has proposed several solutions aiming to equip the neural network with the ability to learn the current task (plasticity) while still achieving high accuracy on the previous tasks (stability). Despite remarkable improvements, the plasticity-stability trade-off is still far from being solved and its underlying mechanism is poorly understood. In this work, we propose Auxiliary Network Continual Learning (ANCL), a novel method that applies an additional auxiliary network which promotes plasticity to the continually learned model which mainly focuses on stability. More concretely, the proposed framework materializes in a regularizer that naturally interpolates between plasticity and stability, surpassing strong baselines on task incremental and class incremental scenarios. Through extensive analyses on ANCL solutions, we identify some essential principles beneath the stability-plasticity trade-off.