mlsanityData to 14 Oct 2026

Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy

arXiv 2310.01334 ↗First posted 2 Oct 2023cs.LG cs.AI cs.CLcited by 108

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse. Therefore, vanilla SMoE models are memory inefficient and non-scalable, especially for resource-constrained downstream scenarios. In this paper, we ask: Can we craft a compact SMoE model by consolidating expert information? What is the best recipe to merge multiple experts into fewer but more knowledgeable experts? Our pilot investigation reveals that conventional model merging methods fail to be effective in such expert merging for SMoE. The potential reasons are: (1) redundant information overshadows critical experts; (2) appropriate neuron permutation for each expert is missing to bring all of them in alignment. To address this, we propose M-SMoE, which leverages routing statistics to guide expert merging. Specifically, it starts with neuron permutation alignment for experts; then, dominant experts and their "group members" are formed; lastly, every expert group is merged into a single expert by utilizing each expert's activation frequency as their weight for merging, thus diminishing the impact of insignificant experts. Moreover, we observed that our proposed merging promotes a low dimensionality in the merged expert's weight space, naturally paving the way for additional compression. Hence, our final method, MC-SMoE (i.e., Merge, then Compress SMoE), further decomposes the merged experts into low-rank and structural sparse alternatives. Extensive experiments across 8 benchmarks validate the effectiveness of MC-SMoE. For instance, our MC-SMoE achieves up to 80% memory and a 20% FLOPs reduction, with virtually no loss in performance.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 65 of its 78 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

65 of 78 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Soft Merging of Experts with Adaptive Routing6 Jun
  2. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers2 Mar
  3. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  10. 2022Editing Models with Task Arithmeticcited by 1,120
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  12. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  14. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  15. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  16. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  17. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  18. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  19. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  20. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  21. 2022Patching open-vocabulary models by interpolating weightscited by 190
  22. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  23. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  24. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  25. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  26. 2022Fusing finetuned models for better pretrainingcited by 110
  27. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  28. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  29. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  30. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  31. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  32. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  33. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  34. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  35. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  36. 2022Re-basin via implicit Sinkhorn differentiationcited by 62
  37. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  38. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 42; 2026: 46.

108 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  3. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  4. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 52
  5. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
  6. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  7. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 37
  8. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 35
  9. 2025Delta Decompression for MoE-based LLMs Compressioncited by 33
  10. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  11. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  12. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
38 more
  1. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  2. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 26
  3. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  4. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  5. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  6. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 20
  7. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 19
  8. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  9. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  10. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  11. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  12. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 14
  13. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 13
  14. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 13
  15. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  16. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  17. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruningcited by 10
  18. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  19. 2025HAMoBE: Hierarchical and Adaptive Mixture of Biometric Experts for Video-based Person ReIDcited by 10
  20. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Mergingcited by 10
  21. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  22. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Expertscited by 9
  23. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  24. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  25. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  26. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 9
  27. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  28. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  29. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 8
  30. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  31. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  32. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Trainingcited by 7
  33. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  34. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 7
  35. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  36. 2024First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Modelscited by 6
  37. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Expertscited by 6
  38. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  2. 2025Improving Routing in Sparse Mixture of Experts with Graph of Tokens
  3. 2026Eigenvectors of Experts are Training-free Non-collapsing Routers
  4. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks
  5. 2025Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
  6. 2026Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts
  7. 2024On the Role of Discrete Representation in Sparse Mixture of Experts
  8. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity