mlsanityData to 14 Oct 2026

Architecture Matters in Continual Learning

arXiv 2202.00275 ↗First posted 1 Feb 2022cs.LG cs.AIcited by 46

Not on the map. Why?

Abstract (arXiv)

A large body of research in continual learning is devoted to overcoming the catastrophic forgetting of neural networks by designing new algorithms that are robust to the distribution shifts. However, the majority of these works are strictly focused on the "algorithmic" part of continual learning for a "fixed neural network architecture", and the implications of using different architectures are mostly neglected. Even the few existing continual learning methods that modify the model assume a fixed architecture and aim to develop an algorithm that efficiently uses the model throughout the learning experience. However, in this work, we show that the choice of architecture can significantly impact the continual learning performance, and different architectures lead to different trade-offs between the ability to remember previous tasks and learning new ones. Moreover, we study the impact of various architectural decisions, and our findings entail best practices and recommendations that can improve the continual learning performance.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2016Wide Residual Networkscited by 3,070
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  8. 2018Group Normalizationcited by 1,524
  9. 2016Progressive Neural Networkscited by 1,361
  10. 2017Continual Learning with Deep Generative Replaycited by 1,158
  11. 2018Experience Replay for Continual Learningcited by 1,095
  12. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
26 more
  1. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  2. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  3. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  4. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  5. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  6. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  7. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  8. 2018How Does Batch Normalization Help Optimization?cited by 357
  9. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  10. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  11. 2020Supermasks in Superpositioncited by 227
  12. 2021Vision Transformers are Robust Learnerscited by 207
  13. 2018Reinforced Continual Learningcited by 186
  14. 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
  15. 2018Understanding Batch Normalizationcited by 159
  16. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  17. 2016Identity Matters in Deep Learningcited by 109
  18. 2017Residual Connections Encourage Iterative Inferencecited by 77
  19. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  20. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  21. 2020Efficient Architecture Search for Continual Learningcited by 34
  22. 2021Kernel Continual Learningcited by 27
  23. 2020The Effectiveness of Memory Replay in Large Scale Continual Learningcited by 26
  24. 2021Task-agnostic Continual Learning with Hybrid Probabilistic Modelscited by 16
  25. 2018Representing smooth functions as compositions of near-identity functions with implications for deep network optimizationcited by 15
  26. 2019Neural Architecture Search for Class-incremental Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 12; 2023: 18; 2024: 8; 2025: 6; 2026: 2.

46 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  3. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  4. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  5. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  6. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifierscited by 67
  7. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  8. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 36
  9. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  10. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  11. 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 29
  12. 2023A Review of Deep Learning for Video Captioningcited by 28
34 more
  1. 2022Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularizationcited by 23
  2. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  3. 2023A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Applicationcited by 21
  4. 2023Is forgetting less a good inductive bias for forward transfer?cited by 18
  5. 2023PromptFusion: Decoupling Stability and Plasticity for Continual Learningcited by 17
  6. 2024Revisiting Neural Networks for Continual Learning: An Architectural Perspectivecited by 17
  7. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  8. 2023Overcoming the Stability Gap in Continual Learningcited by 13
  9. 2024Continual Learning with Weight Interpolationcited by 13
  10. 2022Continual Learning by Modeling Intra-Class Variationcited by 12
  11. 2024On the Diminishing Returns of Width for Continual Learningcited by 12
  12. 2024Random Representations Outperform Online Continually Learned Representationscited by 11
  13. 2023Elephant Neural Networks: Born to Be a Continual Learnercited by 10
  14. 2022Memory Population in Continual Learning via Outlier Eliminationcited by 9
  15. 2023Heterogeneous Continual Learningcited by 6
  16. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  17. 2022Engineering flexible machine learning systems by traversing functionally-invariant pathscited by 4
  18. 2022Exploring the Intersection between Neural Architecture Search and Continual Learningcited by 4
  19. 20223rd Continual Learning Workshop Challenge on Egocentric Category and Instance Level Object Understandingcited by 3
  20. 2023Effects of Architectures on Continual Semantic Segmentationcited by 3
  21. 2025Continual learning for rotating machinery fault diagnosis with cross-domain environmental and operational variationscited by 3
  22. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 3
  23. 2023Efficiently Upgrading Multilingual Machine Translation Models to Support More Languagescited by 2
  24. 2024Benchmarking Sensitivity of Continual Graph Learning for Skeleton-Based Action Recognitioncited by 2
  25. 2024Improving Continual Learning Performance and Efficiency with Auxiliary Classifierscited by 2
  26. 2025Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learningcited by 2
  27. 2022Increasing Depth of Neural Networks for Life-long Learningcited by 1
  28. 2023An Intentional Forgetting-Driven Self-Healing Method For Deep Reinforcement Learning Systemscited by 1
  29. 2023What Can AutoML Do For Continual Learning?cited by 1
  30. 2025A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initializationcited by 1
  31. 2023Dissecting Continual Learning a Structural and Data Analysiscited by 0
  32. 2023Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learningcited by 0
  33. 2026Grow-Prune-Freeze Networks: Adaptive & Continual Learning Technique for Olfactory Navigationcited by 0
  34. 2026What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentationcited by 0