mlsanityData to 7 Oct 2026

Continual Learning Beyond a Single Model

arXiv 2202.09826 ↗First posted 20 Feb 2022cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

A growing body of research in continual learning focuses on the catastrophic forgetting problem. While many attempts have been made to alleviate this problem, the majority of the methods assume a single model in the continual learning setup. In this work, we question this assumption and show that employing ensemble models can be a simple yet effective method to improve continual performance. However, ensembles' training and inference costs can increase significantly as the number of models grows. Motivated by this limitation, we study different ensemble models to understand their benefits and drawbacks in continual learning scenarios. Finally, to overcome the high compute cost of ensembles, we leverage recent advances in neural network subspace to propose a computationally cheap algorithm with similar runtime to a single model yet enjoying the performance benefits of ensembles.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 20 Feb 2022.

Builds on

36 of 52 references.

References · 36

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017Continual Learning with Deep Generative Replaycited by 1,152
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  11. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
24 more
  1. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  2. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  3. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  4. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  5. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  6. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  7. 2021Gradient Projection Memory for Continual Learningcited by 338
  8. 2017Variational Continual Learningcited by 335
  9. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  10. 2020Supermasks in Superpositioncited by 223
  11. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  12. 2020Training independent subnetworks for robust predictioncited by 177
  13. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  14. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 121
  15. 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
  16. 2021Learning Neural Network Subspacescited by 82
  17. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  18. 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
  19. 2020Dropout as an Implicit Gating Mechanism For Continual Learningcited by 24
  20. 2021On Anytime Learning at Macroscalecited by 20
  21. 2020Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processescited by 17
  22. 2021Continual Learning in Deep Networks: an Analysis of the Last Layercited by 17
  23. 2021Task-agnostic Continual Learning with Hybrid Probabilistic Modelscited by 16
  24. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 5; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  2. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  3. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  4. 2024Divide and not forget: Ensemble of selectively trained experts in Continual Learningcited by 38
  5. 2024Learning Equi-angular Representations for Online Continual Learningcited by 15
  6. 2023Improving Online Continual Learning Performance and Stability with Temporal Ensemblescited by 14
  7. 2024Continual Learning in the Presence of Repetitioncited by 10
  8. 2025RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detectioncited by 5
  9. 2024A streamlined Approach to Multimodal Few-Shot Class Incremental Learning for Fine-Grained Datasetscited by 4
  10. 2025Continual learning for rotating machinery fault diagnosis with cross-domain environmental and operational variationscited by 3
  11. 2025Boosting-inspired online learning with transfer for railway maintenancecited by 2
  12. 2023A Self-Commissioning Edge Computing Method for Data-Driven Anomaly Detection in Power Electronic Systemscited by 1
6 more
  1. 2024Forward-Backward Knowledge Distillation for Continual Clusteringcited by 1
  2. 2025CADE: Continual Weakly-supervised Video Anomaly Detection with Ensemblescited by 1
  3. 2024Liquid Ensemble Selection for Continual Learningcited by 0
  4. 2024Expert Routing with Synthetic Data for Continual Learningcited by 0
  5. 2025Task-conditioned Ensemble of Expert Models for Continuous Learningcited by 0
  6. 2026FreeMOCA: Memory-Free Continual Learning for Malicious Code Analysiscited by 0