mlsanityData to 7 Oct 2026

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

arXiv 2402.14800 ↗First posted 22 Feb 2024cs.CL cs.AI cs.LGcited by 140

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

A pivotal advancement in the progress of large language models (LLMs) is the emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs, MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. Different from previous weight pruning methods that rely on specifically designed hardware, this paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques. Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks. Extensive experiments show that our proposed methods can simultaneously reduce model sizes and increase the inference speed, while maintaining satisfactory performance. Data and code will be available at https://github.com/Lucky-Lance/Expert_Sparsity.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 26 of its 37 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

26 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  10. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  11. 2021Accelerating Sparse Deep Neural Networkscited by 213
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
11 more
  1. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  2. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  3. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  4. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  5. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  6. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
  7. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  8. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  9. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  10. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 28
  11. 2023Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 51; 2026: 68.

140 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  3. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  4. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  5. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  6. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  7. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  8. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  9. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  10. 2025Advancing Expert Specialization for Better MoEcited by 37
  11. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  12. 2025Delta Decompression for MoE-based LLMs Compressioncited by 32
38 more
  1. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 31
  2. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  3. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  4. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 26
  5. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  6. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  7. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  8. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
  9. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  10. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  11. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  12. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  13. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  14. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inferencecited by 14
  15. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Expertscited by 14
  16. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  17. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  18. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 12
  19. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 12
  20. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  21. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  22. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
  23. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruningcited by 10
  24. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  25. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  26. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  27. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  28. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 8
  29. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  30. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 8
  31. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  32. 2026XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inferencecited by 7
  33. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  34. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 6
  35. 2025Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retrainingcited by 6
  36. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 6
  37. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 5
  38. 2025PiKV: KV Cache Management System for Mixture of Expertscited by 5

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  2. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  3. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  4. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  5. 2025Mixture of Experts in Large Language Models
  6. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  7. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  8. 2026Higher-order pruning of experts in mixture-of-experts language models