mlsanityData to 14 Oct 2026

Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations

arXiv 2504.05586 ↗First posted 8 Apr 2025cs.LG cs.AIcited by 20

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks. However, vanilla SMoEs have issues such as expert redundancy and heavy memory requirements, making them inefficient and non-scalable, especially for resource-constrained scenarios. Expert-level sparsification of SMoEs involves pruning the least important experts to address these limitations. In this work, we aim to address three questions: (1) What is the best recipe to identify the least knowledgeable subset of experts that can be dropped with minimal impact on performance? (2) How should we perform expert dropping (one-shot or iterative), and what correction measures can we undertake to minimize its drastic impact on SMoE subnetwork capabilities? (3) What capabilities of full-SMoEs are severely impacted by the removal of the least dominant experts, and how can we recover them? Firstly, we propose MoE Experts Compression Suite (MC-Suite), which is a collection of some previously explored and multiple novel recipes to provide a comprehensive benchmark for estimating expert importance from diverse perspectives, as well as unveil numerous valuable insights for SMoE experts. Secondly, unlike prior works with a one-shot expert pruning approach, we explore the benefits of iterative pruning with the re-estimation of the MC-Suite criterion. Moreover, we introduce the benefits of task-agnostic fine-tuning as a correction mechanism during iterative expert dropping, which we term MoE Lottery Subnetworks. Lastly, we present an experimentally validated conjecture that, during expert dropping, SMoEs' instruction-following capabilities are predominantly hurt, which can be restored to a robust level subject to external augmentation of instruction-following capabilities using k-shot examples and supervised fine-tuning.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 50 of its 62 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

50 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts7 Apr
  2. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts1 Jun

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2024Mixtral of Expertscited by 1,823
  11. 2020Knowledge Distillation: A Surveycited by 1,603
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
36 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  6. 2019Importance Estimation for Neural Network Pruningcited by 541
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  8. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  9. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  10. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  11. 2021Accelerating Sparse Deep Neural Networkscited by 223
  12. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  13. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  14. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  15. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  16. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  17. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  18. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  19. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  20. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  21. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  22. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  23. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  24. 2022Structural Pruning via Latency-Saliency Knapsackcited by 41
  25. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  26. 2022Is a Modular Architecture Enough?cited by 36
  27. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  28. 2018Coreset-Based Neural Network Compressioncited by 27
  29. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  30. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  31. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  32. 2023Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Modelscited by 20
  33. 2024Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruningcited by 20
  34. 2024MLP Can Be A Good Transformer Learnercited by 17
  35. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  36. 2025IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretrainingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 17.

20 papers.

Sort:

By area:

  1. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  2. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  3. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  4. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  5. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  6. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  7. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  8. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  9. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  10. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Modelscited by 2
  11. 2026From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Modelscited by 2
  12. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
8 more
  1. 2026Higher-order pruning of experts in mixture-of-experts language modelscited by 1
  2. 2026When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruningcited by 0
  3. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  4. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0
  5. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0
  6. 2026SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Servingcited by 0
  7. 2026BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decodingcited by 0
  8. 2026Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  2. 2023Improving Expert Specialization in Mixture of Experts
  3. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  4. 2025Sigma-MoE-Tiny Technical Report
  5. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  6. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  7. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  8. 2026MoE Lens -- An Expert Is All You Need