mlsanityData to 7 Oct 2026

Model Zoo: A Growing "Brain" That Learns Continually

arXiv 2106.03027 ↗First posted 6 Jun 2021cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

This paper argues that continual learning methods can benefit by splitting the capacity of the learner across multiple models. We use statistical learning theory and experimental analysis to show how multiple tasks can interact with each other in a non-trivial fashion when a single model is trained on them. The generalization error on a particular task can improve when it is trained with synergistic tasks, but can also deteriorate when trained with competing tasks. This theory motivates our method named Model Zoo which, inspired from the boosting literature, grows an ensemble of small models, each of which is trained during one episode of continual learning. We demonstrate that Model Zoo obtains large gains in accuracy on a variety of continual learning benchmark problems. Code is available at https://github.com/grasp-lyrl/modelzoo_continual.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 48 of its 79 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

48 of 79 references.

References · 48

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Wide Residual Networkscited by 3,057
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  4. 2016Matching Networks for One Shot Learningcited by 2,246
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2017Continual Learning with Deep Generative Replaycited by 1,152
  8. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  11. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  12. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
36 more
  1. 2019Three scenarios for continual learningcited by 610
  2. 2017Learning multiple visual domains with residual adapterscited by 553
  3. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  4. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  5. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  6. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  7. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  8. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  9. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  10. 2017Variational Continual Learningcited by 335
  11. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  12. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  13. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
  14. 2018Reinforced Continual Learningcited by 186
  15. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  16. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  17. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  18. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 158
  19. 2018Towards Robust Evaluations of Continual Learningcited by 148
  20. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  21. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  22. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  23. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  24. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 121
  25. 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 113
  26. 2020Generative Feature Replay For Class-Incremental Learningcited by 109
  27. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 104
  28. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  29. 2020On the Value of Target Data in Transfer Learningcited by 62
  30. 2019Recurrence is required to capture the representational dynamics of the human visual systemcited by 58
  31. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
  32. 2019A Unifying Bayesian View of Continual Learningcited by 38
  33. 2020A No-Free-Lunch Theorem for MultiTask Learningcited by 28
  34. 2021Encoders and Ensembles for Task-Free Continual Learningcited by 17
  35. 2016Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Datacited by 9
  36. 2021TAG: Task-based Accumulated Gradients for Lifelong learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 14; 2024: 12; 2025: 10; 2026: 5.

53 papers.

Sort:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2025SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learningcited by 60
  4. 2023Continual Learning: Applications and the Road Forwardcited by 59
  5. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  6. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  7. 2023Continual Learning on Dynamic Graphs via Parameter Isolationcited by 35
  8. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  9. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 32
  10. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  11. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  12. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 17
38 more
  1. 2022An Evolutionary Approach to Dynamic Introduction of Tasks in Large-scale Multitask Learning Systemscited by 16
  2. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  3. 2022Provable and Efficient Continual Representation Learningcited by 14
  4. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 13
  5. 2024Provable Contrastive Continual Learningcited by 13
  6. 2022A Continual Development Methodology for Large-scale Multitask Dynamic ML Systemscited by 9
  7. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  8. 2024Predicting the Susceptibility of Examples to Catastrophic Forgettingcited by 7
  9. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  10. 2022Prospective Learning: Principled Extrapolation to the Futurecited by 6
  11. 2022A picture of the space of typical learnable taskscited by 6
  12. 2023Restricted Orthogonal Gradient Projection for Continual Learningcited by 6
  13. 2024Vision-Language Navigation with Continual Learningcited by 6
  14. 2023Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Modelscited by 4
  15. 2024Many Perception Tasks are Highly Redundant Functions of their Input Datacited by 4
  16. 2024Prospective Learning: Learning for a Dynamic Futurecited by 4
  17. 2025Online time series prediction using feature adjustmentcited by 4
  18. 2022Engineering flexible machine learning systems by traversing functionally-invariant pathscited by 3
  19. 2023Provable Pathways: Learning Multiple Tasks over Multiple Pathscited by 3
  20. 2025Continual learning for rotating machinery fault diagnosis with cross-domain environmental and operational variationscited by 3
  21. 2022Deep Reference Priors: What is the best way to pretrain a model?cited by 2
  22. 2022A Multiagent Framework for the Asynchronous and Collaborative Extension of Multitask ML Systemscited by 2
  23. 2023Active Representation Learning for General Task Space with Applications in Roboticscited by 2
  24. 2023Adapting Machine Learning Diagnostic Models to New Populations Using a Small Amount of Data: Results from Clinical Neurosciencecited by 2
  25. 2025Boosting-inspired online learning with transfer for railway maintenancecited by 2
  26. 2025IRS: Incremental Relationship-guided Segmentation for Digital Pathologycited by 2
  27. 2022DIRA: Dynamic Domain Incremental Regularised Adaptationcited by 1
  28. 2023Multipath agents for modular multitask ML systemscited by 1
  29. 2025Graceful Forgetting in Generative Language Modelscited by 1
  30. 2025Prospective Learning in Retrospectcited by 1
  31. 2026Attesting Model Lineage by Consisted Knowledge Evolution with Fine-Tuning Trajectorycited by 1
  32. 2023Context selectivity with dynamic availability enables lifelong continual learningcited by 0
  33. 2024Lifelong Learning Using a Dynamically Growing Tree of Sub-networks for Domain Generalization in Video Object Segmentationcited by 0
  34. 2024Hyper-Representations: Learning from Populations of Neural Networkscited by 0
  35. 2024Low-Rank Filtering and Smoothing for Sequential Deep Learningcited by 0
  36. 2025Task-conditioned Ensemble of Expert Models for Continuous Learningcited by 0
  37. 2025High-dimensional Asymptotics of Generalization Performance in Continual Ridge Regressioncited by 0
  38. 2026Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weightscited by 0

50 most cited of 53 All 53 (JSON)