# SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs

arXiv: https://arxiv.org/abs/2606.09886 · HTML: https://mlsanity.com/paper/2606.09886/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 30 of 30)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation](https://mlsanity.com/paper/1903.10992/) (2019)
- [Neuron Shapley: Discovering the Responsible Neurons](https://mlsanity.com/paper/2002.09815/) (2020)
- [HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization](https://mlsanity.com/paper/2402.16694/) (2024)
- [Task-Specific Expert Pruning for Sparse Mixture-of-Experts](https://mlsanity.com/paper/2206.00277/) (2022)
- [REAP the Experts: Why Pruning Prevails for One-Shot MoE compression](https://mlsanity.com/paper/2510.13999/) (2025)
- [QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models](https://mlsanity.com/paper/2310.16795/) (2023)
- [SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts](https://mlsanity.com/paper/2404.05089/) (2024)
- [Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2407.09590/) (2024)
- [MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance](https://mlsanity.com/paper/2505.03804/) (2025)
- [Data Shapley Valuation for Efficient Batch Active Learning](https://mlsanity.com/paper/2104.08312/) (2021)
- [DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning](https://mlsanity.com/paper/2509.16105/) (2025)
- [One Student Knows All Experts Know: From Sparse to Dense](https://mlsanity.com/paper/2201.10890/) (2022)
- [Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs](https://mlsanity.com/paper/2509.10377/) (2025)
- [A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts](https://mlsanity.com/paper/2405.16646/) (2024)
- [Unveiling Super Experts in Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2507.23279/) (2025)
- [Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations](https://mlsanity.com/paper/2504.06792/) (2025)
- [Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning](https://mlsanity.com/paper/2412.00069/) (2024)
- [CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis](https://mlsanity.com/paper/2508.02322/) (2025)
- [MoE Pathfinder: Trajectory-driven Expert Pruning](https://mlsanity.com/paper/2512.18425/) (2025)
- [Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models](https://mlsanity.com/paper/2504.12359/) (2025)
- [Rethinking the Knowledge Distillation From the Perspective of Model Calibration](https://mlsanity.com/paper/2111.01684/) (2021)
- [SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models](https://mlsanity.com/paper/2503.07605/) (2025)
- [Enhancing Interpretability for Vision Models via Shapley Value Optimization](https://mlsanity.com/paper/2512.14354/) (2025)

## Built on by (most cited first, 3 of 3)

- [When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning](https://mlsanity.com/paper/2608.02940/) (2026)
- [RAZOR: Pruning Replaceable Experts in LLMs](https://mlsanity.com/paper/2609.30465/) (2026)
- [DIET: Deletion-response Expert Trimming for Video Diffusion Transformers](https://mlsanity.com/paper/2609.37829/) (2026)

## Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) large language models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall: the full expert pool must remain resident to support token-dependent routing. Expert pruning is a direct remedy, but prior criteria typically score experts independently and overlook that MoE inference is inherently coalitional, where outputs arise from routed top-k expert combinations. We propose SHAPE, a task-driven pruning framework that explicitly models intra-layer expert cooperation. SHAPE formulates routing traces on a small calibration set as an empirical cooperative game and assigns interaction-aware expert values via a Shapley-style attribution over observed top-k coalitions, enabling the identification of experts that are essential for high-utility collaborations rather than merely frequent. To preserve MoE topology under a global pruning budget, SHAPE further introduces a quality-coverage selection rule that retains, in each layer, the minimal expert subset covering an α fraction of non-negative Shapley mass, while using bisection to match a target keep rate. Experiments on three modern MoE backbones (Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite) across diverse benchmarks show that SHAPE consistently improves robustness over global and layer-wise pruning variants, maintaining competitive accuracy under 20% and 40% expert pruning without additional training and delivering clear reductions in peak GPU memory footprint. The open-source code is available at https://github.com/Alizen-1009/Shapley-Moe.