mlsanityData to 7 Oct 2026

Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques

arXiv 2406.02500 ↗First posted 4 Jun 2024cs.LG cs.AIcited by 24

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world deployment. The Mixture of Experts (MoE) architecture offers a promising solution by dynamically selecting and activating only a subset of experts during inference, thus substantially reducing computational costs while preserving high performance. Despite these benefits, MoE introduces new inefficiencies, such as excessive parameters and communication overhead. In this work, we present a holistic study of compression techniques for Mixture of Experts to enhance both efficiency and scalability. While recent efforts have focused on Expert Trimming, which reduces the number of experts, these approaches still suffer from considerable communication and computational costs. To address this, we propose more aggressive strategies, such as Layer Drop, which removes entire MoE layers, and Block Drop, which eliminates transformer blocks. Surprisingly, these aggressive pruning techniques not only preserve model performance but also substantially improve computation and memory efficiency. Furthermore, beyond Expert Trimming, we also introduce Expert Slimming, which compresses individual experts to further boost performance and can be seamlessly integrated with Expert Trimming. Extensive experimental results demonstrate the effectiveness of our proposed methods-Layer Drop and Block Drop-along with the comprehensive recipe that integrates Expert Slimming and Expert Trimming, achieving a 6.05x speedup with 77.1% reduced memory usage while maintaining over 92% of performance on Mixtral-8x7B. Our code is released at https://github.com/CASE-Lab-UMD/Unified-MoE-Compression.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 47 of its 58 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

47 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  2. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  4. 2024Mixtral of Experts8 Jan
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
28 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  8. 2023Instruction Tuning with GPT-4cited by 676
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  11. 2021A White Paper on Neural Network Quantizationcited by 453
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  13. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  14. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  15. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  16. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  17. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  18. 2021Hash Layers For Large Sparse Modelscited by 236
  19. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  20. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  21. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 122
  22. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  23. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  24. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  25. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Servicescited by 37
  26. 2022Dual Lottery Ticket Hypothesiscited by 27
  27. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  28. 2022PAD-Net: An Efficient Framework for Dynamic Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 13.

24 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  3. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  4. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  5. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  6. 2025Delta Decompression for MoE-based LLMs Compressioncited by 32
  7. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  8. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  9. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  10. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 8
  11. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  12. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
12 more
  1. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  2. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  3. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2
  4. 2026Demystifying When Pruning Works via Representation Hierarchiescited by 2
  5. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Modelscited by 2
  6. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  7. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillationcited by 0
  8. 2025Systematic Optimization of Open Source Large Language Models for Mathematical Reasoningcited by 0
  9. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Expertscited by 0
  10. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  11. 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Modelscited by 0
  12. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  2. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  3. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  4. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  5. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Models
  6. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
  7. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
  8. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts