mlsanityData to 14 Oct 2026

Task-Specific Expert Pruning for Sparse Mixture-of-Experts

arXiv 2206.00277 ↗First posted 1 Jun 2022cs.LG cs.AIcited by 72

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pre-training and has achieved promising results due to its model capacity. However, with trillions of parameters, MoE is hard to be deployed on cloud or mobile environment. The inference of MoE requires expert parallelism, which is not hardware-friendly and communication expensive. Especially for resource-limited downstream tasks, such sparse structure has to sacrifice a lot of computing efficiency for limited performance gains. In this work, we observe most experts contribute scarcely little to the MoE fine-tuning and inference. We further propose a general method to progressively drop the non-professional experts for the target downstream task, which preserves the benefits of MoE while reducing the MoE model into one single-expert dense model. Our experiments reveal that the fine-tuned single-expert model could preserve 99.3% benefits from MoE across six different types of tasks while enjoying 2x inference speed with free communication cost.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 22 of its 32 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

22 of 32 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
10 more
  1. 2018Neural Network Acceptability Judgmentscited by 958
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  5. 2021Hash Layers For Large Sparse Modelscited by 237
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  7. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  8. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  9. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  10. 2021Tricks for Training Sparse Translation Modelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 15; 2025: 20; 2026: 29.

72 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  2. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  3. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  4. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  6. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  7. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  8. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  9. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 53
  10. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  11. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  12. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
38 more
  1. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 37
  2. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  3. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 35
  4. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  5. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  6. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  7. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 20
  8. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 19
  9. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  10. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  11. 2022MoEC: Mixture of Expert Clusterscited by 14
  12. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  13. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 14
  14. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 13
  15. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  16. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Expertscited by 9
  17. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  18. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  19. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 8
  20. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  21. 2024Multimodal Clinical Trial Outcome Prediction with Large Language Modelscited by 7
  22. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Trainingcited by 7
  23. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Modelscited by 7
  24. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  25. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  26. 2023Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulationcited by 5
  27. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurancecited by 5
  28. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  29. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
  30. 2026Expert Divergence Learning for MoE-based Language Modelscited by 4
  31. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilitiescited by 3
  32. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitutioncited by 3
  33. 2026Component-Level Lesioning of Language Models Reveals Clinically Aligned Aphasia Phenotypescited by 3
  34. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  35. 2025MoENAS: Mixture-of-Expert based Neural Architecture Search for jointly Accurate, Fair, and Robust Edge Deep Neural Networkscited by 2
  36. 2026DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networkscited by 2
  37. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 2
  38. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?cited by 2

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
  2. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
  3. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
  4. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  5. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  6. 2026MoE Lens -- An Expert Is All You Need
  7. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  8. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models