mlsanityData to 7 Oct 2026

Expert Gate: Lifelong Learning with a Network of Experts

arXiv 1611.06194 ↗First posted 18 Nov 2016cs.CV cs.AI stat.MLcited by 369

Not on the map. Why?

Cited by 28 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper we introduce a model of lifelong learning, based on a Network of Experts. New tasks / experts are learned and added to the model sequentially, building on what was learned before. To ensure scalability of this process,data from previous tasks cannot be stored and hence is not available when learning a new task. A critical issue in such context, not addressed in the literature so far, relates to the decision which expert to deploy at test time. We introduce a set of gating autoencoders that learn a representation for the task at hand, and, at test time, automatically forward the test sample to the relevant expert. This also brings memory efficiency as only one expert network has to be loaded into memory at any given time. Further, the autoencoders inherently capture the relatedness of one task to another, based on which the most relevant prior model to be used for training a new expert, with finetuning or learning without-forgetting, can be selected. We evaluate our method on image classification and video prediction problems.

Key facts. Cited by 369 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 0; 2020: 1; 2021: 42; 2022: 67; 2023: 88; 2024: 76; 2025: 58; 2026: 36.

369 papers.

Sort:

By area:

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  4. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  5. 2023Class-Incremental Learning: A Surveycited by 209
  6. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  9. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  10. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  11. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  12. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
38 more
  1. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  2. 2022Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolutioncited by 96
  3. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  4. 2023Modular Deep Learningcited by 82
  5. 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
  6. 2022A Theoretical Study on Solving Continual Learningcited by 78
  7. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 74
  8. 2024Continual Learning and Catastrophic Forgettingcited by 73
  9. 2024Mixture of A Million Expertscited by 70
  10. 2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
  11. 2023RMM: Reinforced Memory Management for Class-Incremental Learningcited by 66
  12. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  13. 2023Can Neural Network Memorization Be Localized?cited by 66
  14. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  15. 2021Continual Learning via Local Module Compositioncited by 61
  16. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  17. 2022Class-Incremental Learning with Strong Pre-trained Modelscited by 60
  18. 2023Continual Detection Transformer for Incremental Object Detectioncited by 59
  19. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  20. 2022An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditionscited by 54
  21. 2021Self-Supervised Training Enhances Online Continual Learningcited by 53
  22. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 53
  23. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  24. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  25. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  26. 2023A Survey on Few-Shot Class-Incremental Learningcited by 48
  27. 2023Dense Network Expansion for Class Incremental Learningcited by 45
  28. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  29. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  30. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  31. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  32. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
  33. 2024Divide and not forget: Ensemble of selectively trained experts in Continual Learningcited by 38
  34. 2024Efficient Dictionary Learning with Switch Sparse Autoencoderscited by 38
  35. 2021Continual Learning for Blind Image Quality Assessmentcited by 36
  36. 2021Multi-View Correlation Distillation for Incremental Object Detectioncited by 36
  37. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 36
  38. 2022Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Taskcited by 35

50 most cited of 369 All 369 (JSON)