mlsanityData to 14 Oct 2026

Does a Global Perspective Help Prune Sparse MoEs Elegantly?

arXiv 2604.06542 ↗First posted 8 Apr 2026cs.CLcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Empirical scaling laws for language models have encouraged the development of ever-larger LLMs, despite their growing computational and memory costs. Sparse Mixture-of-Experts (MoEs) offer a promising alternative by activating only a subset of experts per forward pass, improving efficiency without sacrificing performance. However, the large number of expert parameters still leads to substantial memory consumption. Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs. We propose GRAPE (Global Redundancy-Aware Pruning of Experts, a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy. Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance. On the three main models reported in the paper, it improves average accuracy over the strongest local baseline by 1.40% on average across pruning settings, with gains of up to 2.45%.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 12 of its 19 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

12 of 19 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning10 Sep
  2. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  3. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  4. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models8 Apr
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb

Other references · 6

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  3. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  4. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 87
  5. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  6. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  2. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  2. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  3. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  4. 2026Higher-order pruning of experts in mixture-of-experts language models
  5. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
  6. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
  7. 2026PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
  8. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques