mlsanityData to 14 Oct 2026

DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning

arXiv 2509.16105 ↗First posted 19 Sep 2025cs.CLcited by 26

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Despite the significant breakthrough of Mixture-of-Experts (MoE), the increasing scale of these MoE models presents huge memory and storage challenges. Existing MoE pruning methods, which involve reducing parameter size with a uniform sparsity across all layers, often lead to suboptimal outcomes and performance degradation due to varying expert redundancy in different MoE layers. To address this, we propose a non-uniform pruning strategy, dubbed Differentiable Expert Pruning (DiEP), which adaptively adjusts pruning rates at the layer level while jointly learning inter-layer importance, effectively capturing the varying redundancy across different MoE layers. By transforming the global discrete search space into a continuous one, our method handles exponentially growing non-uniform expert combinations, enabling adaptive gradient-based pruning. Extensive experiments on five advanced MoE models demonstrate the efficacy of our method across various NLP tasks. Notably, DiEP retains around 92% of original performance on Mixtral 8×7B with only half the experts, outperforming other pruning methods by up to 7.1% on the challenging MMLU dataset.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 39 of its 51 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

39 of 51 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Delta Decompression for MoE-based LLMs Compression24 Feb
  2. 2024Qwen2.5 Technical Report19 Dec
  3. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition1 Nov
  4. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  5. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  6. 2024A Survey on Mixture of Experts in Large Language Models26 Jun
  7. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  8. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts20 Feb
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  10. 2024Mixtral of Experts8 Jan
  11. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer15 Oct
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
1 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
14 more
  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  2. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  3. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  4. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  5. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  6. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 64
  7. 2020Discrimination-aware Network Pruning for Deep Model Compressioncited by 51
  8. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  9. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  10. 2016Convolutional Neural Fabricscited by 32
  11. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  12. 2023Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?cited by 20
  13. 2023Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulatorscited by 9
  14. 2017Connectivity Learning in Multi-Branch Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 25.

26 papers.

Sort:

By area:

  1. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  2. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  3. 2026XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inferencecited by 7
  4. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  5. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  6. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Expertscited by 3
  7. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  8. 2026TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesiscited by 1
  9. 2026UniPool: A Globally Shared Expert Pool for Mixture-of-Expertscited by 1
  10. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1
  11. 2026ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compressioncited by 1
  12. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
14 more
  1. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Expertscited by 0
  2. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  3. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Modelscited by 0
  4. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  5. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0
  6. 2026MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Modelscited by 0
  7. 2026Dense to MoE Adaptation for Compact Vision Language Action Policiescited by 0
  8. 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMscited by 0
  9. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuitcited by 0
  10. 2026DIET: Deletion-response Expert Trimming for Video Diffusion Transformerscited by 0
  11. 2026Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimizationcited by 0
  12. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Modelscited by 0
  13. 2026PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compressioncited by 0
  14. 2026SLEM: Spectrum-Aware Latent Expert Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  2. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  3. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  4. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  5. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
  6. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
  7. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
  8. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE