mlsanityData to 7 Oct 2026

Learning More Generalized Experts by Merging Experts in Mixture-of-Experts

arXiv 2405.11530 ↗First posted 19 May 2024cs.LGcited by 12

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

We observe that incorporating a shared layer in a mixture-of-experts can lead to performance degradation. This leads us to hypothesize that learning shared features poses challenges in deep learning, potentially caused by the same feature being learned as various different features. To address this issue, we track each expert's usage frequency and merge the two most frequently selected experts. We then update the least frequently selected expert using the combination of experts. This approach, combined with the subsequent learning of the router's expert selection, allows the model to determine if the most frequently selected experts have learned the same feature differently. If they have, the combined expert can be further trained to learn a more general feature. Consequently, our algorithm enhances transfer learning and mitigates catastrophic forgetting when applied to multi-domain task incremental learning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 38 of its 47 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

38 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters18 Mar

Other references · 37

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2016Learning without Forgettingcited by 2,465
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
25 more
  1. 2017Deep Mutual Learningcited by 621
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2023From Sparse to Soft Mixtures of Expertscited by 245
  5. 2021Hash Layers For Large Sparse Modelscited by 236
  6. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
  7. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  8. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  9. 2022Patching open-vocabulary models by interpolating weightscited by 187
  10. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  11. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  12. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  13. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  14. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  15. 2022Fusing finetuned models for better pretrainingcited by 110
  16. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  17. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  18. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 49
  19. 2023Dense Network Expansion for Class Incremental Learningcited by 45
  20. 2024Divide and not forget: Ensemble of selectively trained experts in Continual Learningcited by 38
  21. 2024Multi-Head Mixture-of-Expertscited by 30
  22. 2020Neural networks with late-phase weightscited by 29
  23. 2023Class Incremental Learning via Likelihood Ratio Based Task Predictioncited by 17
  24. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  25. 2024Diverse Feature Learning by Self-distillation and Resetcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 3.

12 papers.

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2025Mixture of Experts in Large Language Modelscited by 15
  4. 2025DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Roboticscited by 8
  5. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusioncited by 3
  6. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  7. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  8. 2024MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translationcited by 0
  9. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  10. 2024Expert Routing with Synthetic Data for Continual Learningcited by 0
  11. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0
  12. 2026MECAIL: Communication-Aware Incremental Learning for Object Detection with 14.6 KB Spatiotemporal Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Theory on Mixture-of-Experts in Continual Learning
  2. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  3. 2023Improving Expert Specialization in Mixture of Experts
  4. 2025Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
  5. 2026MoRE: Mixture of Reused Experts
  6. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners
  7. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
  8. 2018Deep Mixture of Experts via Shallow Embedding