mlsanityData to 7 Oct 2026

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

arXiv 2407.09590 ↗First posted 12 Jul 2024cs.CL cs.LGcited by 36

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

By increasing model parameters but activating them sparsely when performing a task, the use of Mixture-of-Experts (MoE) architecture significantly improves the performance of Large Language Models (LLMs) without increasing the inference cost. However, the memory consumption due to the growing number of experts presents a challenge to the deployment of these models in many real world settings. Our empirical study reveals that some experts encode redundant knowledge during pre-training. We thus propose a method of grouping and pruning similar experts to improve the model's parameter efficiency. We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures, including Mixtral, Deepseek-MoE, and Qwen. The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks. We will release our code to facilitate future research.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 56 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

30 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques4 Jun
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 26

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  10. 2018Rethinking the Value of Network Pruningcited by 595
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  12. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
14 more
  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  2. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  3. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  4. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  5. 2019Pruning from Scratchcited by 72
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  7. 2024LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insightscited by 65
  8. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  9. 2020Discrimination-aware Network Pruning for Deep Model Compressioncited by 50
  10. 2021When to Prune? A Policy towards Early Structural Pruningcited by 38
  11. 2019Robust Learning with the Hilbert-Schmidt Independence Criterioncited by 36
  12. 2020Empirical Studies on the Properties of Linear Regions in Deep Neural Networkscited by 36
  13. 2018Structured Pruning of Neural Networks with Budget-Aware Regularizationcited by 26
  14. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 20.

36 papers.

Sort:

By area:

  1. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  4. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  5. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  6. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  7. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  8. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  9. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 7
  10. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  11. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Modelscited by 6
  12. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
24 more
  1. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  2. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  3. 2025Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learningcited by 4
  4. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMscited by 4
  5. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  6. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMscited by 3
  7. 2025Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Modelscited by 3
  8. 2025Learning Compact Vision Tokens for Efficient Large Multimodal Modelscited by 3
  9. 2026SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Modelscited by 3
  10. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  11. 2025Model Assembly Learning with Heterogeneous Layer Weight Mergingcited by 2
  12. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?cited by 2
  13. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  14. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
  15. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  16. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  17. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  18. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  19. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
  20. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decodingcited by 1
  21. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  22. 2026SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignmentcited by 0
  23. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  24. 2026DeaMoE: Efficient MoE Structure for Fast Small-Batch Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  2. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  3. 2026Higher-order pruning of experts in mixture-of-experts language models
  4. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
  5. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
  6. 2025Mixture of Experts in Large Language Models
  7. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
  8. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts