mlsanityData to 7 Oct 2026

Merging Experts into One: Improving Computational Efficiency of Mixture of Experts

arXiv 2310.09832 ↗First posted 15 Oct 2023cs.CLcited by 36

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of growing computational cost. Although a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters (e.g., one expert) for each input, its computation escalates significantly if increasing the number of activated experts, limiting its practical utility. Can we retain the advantages of adding more experts without substantially increasing the computational costs? In this paper, we first demonstrate the superiority of selecting multiple experts and then propose a computation-efficient approach called textbftextttMerging Experts into One (MEO), which reduces the computation cost to that of a single expert. Extensive experiments show that MEO significantly improves computational efficiency, e.g., FLOPS drops from 72.0G of vanilla MoE to 28.6G (MEO). Moreover, we propose a token-level attention block that further enhances the efficiency and performance of token-level MEO, e.g., 83.3% (MEO) vs. 82.6% (vanilla MoE) average score on the GLUE benchmark. Our code will be released upon acceptance. Code will be released at: urlhttps://github.com/Shwai-He/MEO.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 33 of its 37 references that are in the map. First posted on arXiv 15 Oct 2023.

Builds on

33 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  3. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
17 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  3. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  4. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  5. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  6. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  7. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  8. 2023Deep Model Fusion: A Surveycited by 89
  9. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  10. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  11. 2022Omni-Dimensional Dynamic Convolutioncited by 55
  12. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  13. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  14. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  15. 2023MerA: Merging Pretrained Adapters For Few-Shot Learningcited by 13
  16. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  17. 2022PAD-Net: An Efficient Framework for Dynamic Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 12; 2026: 14.

36 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  3. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  4. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agentscited by 30
  5. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
  6. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  7. 2024Multi-granular Adversarial Attacks against Black-box Neural Ranking Modelscited by 14
  8. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  9. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  10. 2024MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Expertscited by 9
  11. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  12. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theorycited by 8
24 more
  1. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Mergingcited by 8
  2. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  3. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  4. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  5. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  6. 2024Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detectioncited by 4
  7. 2025CAMEx: Curvature-aware Merging of Expertscited by 4
  8. 2025MASS: MoErging through Adaptive Subspace Selectioncited by 4
  9. 2023RIGHT: Retrieval-augmented Generation for Mainstream Hashtag Recommendationcited by 2
  10. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 2
  11. 2025Improving Recursive Transformers with Mixture of LoRAscited by 2
  12. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  13. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  14. 2026UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Modelscited by 2
  15. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inferencecited by 1
  16. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  17. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 0
  18. 2025Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inferencecited by 0
  19. 2026Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimizationcited by 0
  20. 2026ExFusion: Efficient Transformer Training via Multi-Experts Fusioncited by 0
  21. 2026Compositional Generalization in Autoregressive Models via Logit Compositioncited by 0
  22. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Expertscited by 0
  23. 2026Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMscited by 0
  24. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  3. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
  4. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  5. 2024Multi-Head Mixture-of-Experts
  6. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  7. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  8. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques