mlsanityData to 14 Oct 2026

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations

arXiv 2504.06792 ↗First posted 9 Apr 2025cs.CL cs.LGcited by 16

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models achieve a favorable trade-off between performance and inference efficiency by activating only a subset of experts. However, the memory overhead of storing all experts remains a major limitation, especially in large-scale MoE models such as DeepSeek-R1(671B). In this study, we investigate domain specialization and expert redundancy in large-scale MoE models and uncover a consistent behavior we term few-shot expert localization, with only a few in-domain demonstrations, the model consistently activates a sparse and stable subset of experts on tasks within the same domain. Building on this observation, we propose a simple yet effective pruning framework, EASY-EP, that leverages a few domain-specific demonstrations to identify and retain only the most relevant experts. EASY-EP comprises two key components: output-aware expert importance assessment and expert-level token contribution estimation. The former evaluates the importance of each expert for the current token by considering the gating scores and L2 norm of the outputs of activated experts, while the latter assesses the contribution of tokens based on representation similarities before and after routed experts. Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and 2.99× throughput under the same memory budget with full model with only half the experts.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 32 of its 42 references that are in the map. First posted on arXiv 9 Apr 2025.

Builds on

32 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy2 Oct
  2. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Experts5 Oct
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 29

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2024Mixtral of Expertscited by 1,823
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
17 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  2. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  3. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  4. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  5. 2023XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameterscited by 92
  6. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 64
  7. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 59
  8. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  9. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  10. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  11. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  12. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  13. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  14. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 25
  15. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  16. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruningcited by 10
  17. 2025Mixture of Tunable Experts -- Behavior Modification of DeepSeek-R1 at Inference Timecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 14.

16 papers.

Sort:

By area:

  1. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  2. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  3. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  4. 2026EMO: Pretraining Mixture of Experts for Emergent Modularitycited by 4
  5. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  6. 2026Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Codingcited by 3
  7. 2025PreMoE: Proactive Inference for Efficient Mixture-of-Expertscited by 1
  8. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  9. 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocationcited by 1
  10. 2026Higher-order pruning of experts in mixture-of-experts language modelscited by 1
  11. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Expertscited by 0
  12. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
4 more
  1. 2026UMoE:Unlocking Every Expert in Domain-Specific Trainingcited by 0
  2. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0
  3. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuitcited by 0
  4. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  2. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  3. 2026It Takes a MAESTRO To Prune Bad Experts
  4. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  5. 2026MoE Lens -- An Expert Is All You Need
  6. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  7. 2026Towards a Statistical Understanding of Mixture-of-Experts
  8. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference