mlsanityData to 14 Oct 2026

Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning

arXiv 2303.09483 ↗First posted 16 Mar 2023cs.LG cs.CVcited by 41

Not on the map. Why?

Abstract (arXiv)

In contrast to the natural capabilities of humans to learn new tasks in a sequential fashion, neural networks are known to suffer from catastrophic forgetting, where the model's performances on old tasks drop dramatically after being optimized for a new task. Since then, the continual learning (CL) community has proposed several solutions aiming to equip the neural network with the ability to learn the current task (plasticity) while still achieving high accuracy on the previous tasks (stability). Despite remarkable improvements, the plasticity-stability trade-off is still far from being solved and its underlying mechanism is poorly understood. In this work, we propose Auxiliary Network Continual Learning (ANCL), a novel method that applies an additional auxiliary network which promotes plasticity to the continually learned model which mainly focuses on stability. More concretely, the proposed framework materializes in a regularizer that naturally interpolates between plasticity and stability, surpassing strong baselines on task incremental and class incremental scenarios. Through extensive analyses on ANCL solutions, we identify some essential principles beneath the stability-plasticity trade-off.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

34 of 52 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2016Learning without Forgettingcited by 2,496
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  8. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  9. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  10. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  11. 2019Large Scale Incremental Learningcited by 732
  12. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
22 more
  1. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  2. 2019Three scenarios for continual learningcited by 614
  3. 2018End-to-End Incremental Learningcited by 597
  4. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  5. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  6. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  7. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  8. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  9. 2018Learning without Memorizingcited by 250
  10. 2017No Fuss Distance Metric Learning using Proxiescited by 226
  11. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  12. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  13. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  14. 2020Adaptive Aggregation Networks for Class-Incremental Learningcited by 149
  15. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  16. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
  17. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  18. 2016Less-forgetting Learning in Deep Neural Networkscited by 97
  19. 2021AFEC: Active Forgetting of Negative Transfer in Continual Learningcited by 73
  20. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  21. 2021Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connectorcited by 34
  22. 2021How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 17; 2025: 13; 2026: 7.

41 papers.

Sort:

By area:

  1. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 60
  2. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  3. 2023Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learningcited by 16
  4. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  5. 2025ProtoDepth: Unsupervised Continual Depth Completion with Prototypescited by 11
  6. 2023BrainWash: A Poisoning Attack to Forget in Continual Learningcited by 9
  7. 2024Statistical Context Detection for Deep Lifelong Reinforcement Learningcited by 7
  8. 2024Continual Audio-Visual Sound Separationcited by 7
  9. 2026Why Fine-Tuning Encourages Hallucinations and How to Fix Itcited by 7
  10. 2024Vision-Language Navigation with Continual Learningcited by 6
  11. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Modelscited by 6
  12. 2023IBCL: Zero-shot Model Generation under Stability-Plasticity Trade-offscited by 5
29 more
  1. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
  2. 2024UnCLe: Benchmarking Unsupervised Continual Learning for Depth Completioncited by 5
  3. 2025Active Learning for Continual Learning: Keeping the Past Alive in the Presentcited by 5
  4. 2025HeaRT: A Hierarchical Circuit Reasoning Tree-Based Agentic Framework for AMS Design Optimizationcited by 5
  5. 2024Step Out and Seek Around: On Warm-Start Training with Incremental Datacited by 4
  6. 2024Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replaycited by 3
  7. 2024Characterizing Continual Learning Scenarios and Strategies for Audio Analysiscited by 3
  8. 2025Divide-and-Conquer for Enhancing Unlabeled Learning, Stability, and Plasticity in Semi-supervised Continual Learningcited by 3
  9. 2026Elastic Weight Consolidation Done Right for Continual Learningcited by 3
  10. 2024Improving Continual Learning Performance and Efficiency with Auxiliary Classifierscited by 2
  11. 2024Continual Learning for Remote Physiological Measurement: Minimize Forgetting and Simplify Inferencecited by 2
  12. 2024An Effective Dynamic Gradient Calibration Method for Continual Learningcited by 2
  13. 2024Mitigating the Stability-Plasticity Dilemma in Adaptive Train Scheduling with Curriculum-Driven Continual DQN Expansioncited by 2
  14. 2025Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticitycited by 2
  15. 2025Flashbacks to Harmonize Stability and Plasticity in Continual Learningcited by 2
  16. 2025InTAct: Interval-based Task Activation Consolidation for Continual Learningcited by 2
  17. 2025The Bayesian Approach to Continual Learning: An Overviewcited by 1
  18. 2025Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learningcited by 1
  19. 2026Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentationcited by 1
  20. 2023Class-Adaptive Sampling Policy for Efficient Continual Learningcited by 0
  21. 2024Fixed Random Classifier Rearrangement for Continual Learningcited by 0
  22. 2024Continual Learning: Less Forgetting, More OOD Generalization via Adaptive Contrastive Replaycited by 0
  23. 2025Towards Robust Incremental Learning under Ambiguous Supervisioncited by 0
  24. 2025Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learningcited by 0
  25. 2025CL-BioGAN: Biologically-Inspired Cross-Domain Continual Learning for Hyperspectral Anomaly Detectioncited by 0
  26. 2026Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacitiescited by 0
  27. 2026HSI Image Enhancement Classification Based on Knowledge Distillation: A Study on Forgettingcited by 0
  28. 2026Task Switching Without Forgetting via Proximal Decouplingcited by 0
  29. 2026Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learningcited by 0