mlsanityData to 7 Oct 2026

MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router

arXiv 2410.12013 ↗First posted 15 Oct 2024cs.CL cs.AI cs.LGcited by 58

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model performance. Motivated by the recent observation of emergent large magnitude features in Large Language Models (LLM) and MoE routing policy, we propose MoE-Pruner, a method that prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron. Our pruning method is one-shot, requiring no retraining or weight updates. We evaluate our method on Mixtral-8x7B and Mixtral-8x22B across multiple language benchmarks. Experimental results show that our pruning method significantly outperforms state-of-the-art LLM pruning methods. Furthermore, our pruned MoE models can benefit from a pretrained teacher model through expert-wise knowledge distillation, improving performance post-pruning. Experimental results demonstrate that the Mixtral-8x7B model with 50% sparsity maintains 99% of the performance of the original model after the expert-wise knowledge distillation.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 60 of its 71 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

60 of 71 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  2. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  3. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques4 Jun
  4. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  8. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models25 Oct
  9. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness3 Oct
  10. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  11. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
1 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2024The Llama 3 Herd of Modelscited by 14,192
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
35 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  13. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  14. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  15. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  16. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  17. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  18. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  19. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  20. 2018Rethinking the Value of Network Pruningcited by 595
  21. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  22. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  23. 2021Accelerating Sparse Deep Neural Networkscited by 213
  24. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  25. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  26. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  27. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  28. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  29. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  30. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  31. 2022Is a Modular Architecture Enough?cited by 35
  32. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  33. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  34. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18
  35. 2024Yuan 2.0-M32: Mixture of Experts with Attention Routercited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 21; 2026: 35.

58 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025Delta Decompression for MoE-based LLMs Compressioncited by 32
  3. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 26
  4. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  5. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  6. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  7. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  8. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  9. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 12
  10. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 12
  11. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  12. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
38 more
  1. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  2. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decodingcited by 9
  3. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  4. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Mergingcited by 8
  5. 2026XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inferencecited by 7
  6. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  7. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Expertscited by 6
  8. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 6
  9. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  10. 2025CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysiscited by 5
  11. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inferencecited by 5
  12. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  13. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilitiescited by 3
  14. 2025HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Spacecited by 3
  15. 2025Collaborative Compression for Large-Scale MoE Deployment on Edgecited by 3
  16. 2025ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorizationcited by 2
  17. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacingcited by 2
  18. 2026Temporally Extended Mixture-of-Experts Modelscited by 2
  19. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Modelscited by 2
  20. 2026From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Modelscited by 2
  21. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  22. 2025MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMscited by 1
  23. 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMscited by 1
  24. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
  25. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  26. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inferencecited by 1
  27. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  28. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1
  29. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  30. 2025Automatic Pruning Discovery for Large Language Modelscited by 0
  31. 2026Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLMcited by 0
  32. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Expertscited by 0
  33. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  34. 2026EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decodingcited by 0
  35. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  36. 2026MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Trainingcited by 0
  37. 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocationcited by 0
  38. 2026PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Compositioncited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  2. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  4. 2025Mixture of Experts in Large Language Models
  5. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
  6. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
  7. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning
  8. 2026Higher-order pruning of experts in mixture-of-experts language models