mlsanityData to 14 Oct 2026

Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective

arXiv 2506.03951 ↗First posted 4 Jun 2025cs.LG cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

The quest for Continual Learning (CL) seeks to empower neural networks with the ability to learn and adapt incrementally. Central to this pursuit is addressing the stability-plasticity dilemma, which involves striking a balance between two conflicting objectives: preserving previously learned knowledge and acquiring new knowledge. While numerous CL methods aim to achieve this trade-off, they often overlook the impact of network architecture on stability and plasticity, restricting the trade-off to the parameter level. In this paper, we delve into the conflict between stability and plasticity at the architectural level. We reveal that under an equal parameter constraint, deeper networks exhibit better plasticity, while wider networks are characterized by superior stability. To address this architectural-level dilemma, we introduce a novel framework denoted Dual-Arch, which serves as a plug-in component for CL. This framework leverages the complementary strengths of two distinct and independent networks: one dedicated to plasticity and the other to stability. Each network is designed with a specialized and lightweight architecture, tailored to its respective objective. Extensive experiments demonstrate that Dual-Arch enhances the performance of existing CL methods while being up to 87% more compact in terms of parameters. Code: https://github.com/byyx666/Dual-Arch.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 51 of its 66 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

51 of 66 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2016Learning without Forgettingcited by 2,496
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2016Progressive Neural Networkscited by 1,361
  10. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  11. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  12. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
38 more
  1. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  2. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  3. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  4. 2019Gradient based sample selection for online continual learningcited by 571
  5. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  6. 2016On the Expressive Power of Deep Neural Networkscited by 306
  7. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  8. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  9. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 229
  10. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  11. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 154
  12. 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 112
  13. 2017Incremental Learning Through Deep Adaptationcited by 108
  14. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  15. 2021PyCIL: A Python Toolbox for Class-Incremental Learningcited by 83
  16. 2016Why Deep Neural Networks for Function Approximation?cited by 82
  17. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
  18. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  19. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  20. 2023PCR: Proxy-based Contrastive Replay for Online Class-Incremental Continual Learningcited by 52
  21. 2019Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replaycited by 51
  22. 2022Architecture Matters in Continual Learningcited by 46
  23. 2021Posterior Meta-Replay for Continual Learningcited by 45
  24. 2024Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networkscited by 43
  25. 2023Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learningcited by 41
  26. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  27. 2024Image Fusion via Vision-Language Modelcited by 40
  28. 2024Make Continual Learning Stronger via C-Flatcited by 35
  29. 2024Resurrecting Old Classes with New Data for Exemplar-Free Continual Learningcited by 33
  30. 2023Improving Plasticity in Online Continual Learning via Collaborative Learningcited by 20
  31. 2023Rethinking Momentum Knowledge Distillation in Online Continual Learningcited by 18
  32. 2024Revisiting Neural Networks for Continual Learning: An Architectural Perspectivecited by 17
  33. 2022SepViT: Separable Vision Transformercited by 16
  34. 2024Hyperparameters in Continual Learning: A Reality Checkcited by 12
  35. 2021Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantizationcited by 11
  36. 2022SYNERgy between SYNaptic consolidation and Experience Replay for general continual learningcited by 10
  37. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  38. 2023MIND: Multi-Task Incremental Network Distillationcited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 8
  2. 2026Continual GUI Agentscited by 7
  3. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Modelscited by 3
  4. 2026Rotation-Preserving Supervised Fine-Tuningcited by 3
  5. 2025C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learningcited by 1
  6. 2026Same Methods, Different Rankings: Trainable Depth as an Evaluation Variable in Continual Learningcited by 1
  7. 2026Continual Learning of Domain-Invariant Representationscited by 1
  8. 2026When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometrycited by 0
  9. 2026Soul Computing: A Theoretical Framework and Technical Architecture for Intelligent Agents with Independent Consciousnesscited by 0
  10. 2026Dimensionality Controls When Modularity Helps in Continual Learningcited by 0
  11. 2026Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensorscited by 0
  12. 2026Continual Learning in Transitioncited by 0
2 more
  1. 2026Sustaining Plasticity via Learnable Wavelet Activations in Continual Learningcited by 0
  2. 2026Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamicscited by 0