# REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

arXiv: https://arxiv.org/abs/2510.13999 · HTML: https://mlsanity.com/paper/2510.13999/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 25 of its 86 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 25 of 25)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [Task-Specific Expert Pruning for Sparse Mixture-of-Experts](https://mlsanity.com/paper/2206.00277/) (2022)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs](https://mlsanity.com/paper/2407.00945/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [Answer Matching Outperforms Multiple Choice for Language Model Evaluation](https://mlsanity.com/paper/2507.02856/) (2025)
- [SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts](https://mlsanity.com/paper/2404.05089/) (2024)
- [Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques](https://mlsanity.com/paper/2406.02500/) (2024)
- [Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations](https://mlsanity.com/paper/2504.05586/) (2025)
- [QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts](https://mlsanity.com/paper/2406.08155/) (2024)
- [The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse](https://mlsanity.com/paper/2410.12766/) (2024)
- [Efficiently Editing Mixture-of-Experts Models with Compressed Experts](https://mlsanity.com/paper/2503.00634/) (2025)

## Built on by (most cited first, 30 of 44)

- [LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts](https://mlsanity.com/paper/2601.18089/) (2026)
- [REAM: Merging Improves Pruning of Experts in LLMs](https://mlsanity.com/paper/2604.04356/) (2026)
- [EvoESAP: Non-Uniform Expert Pruning for Sparse MoE](https://mlsanity.com/paper/2603.06003/) (2026)
- [MoE Pathfinder: Trajectory-driven Expert Pruning](https://mlsanity.com/paper/2512.18425/) (2025)
- [SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training](https://mlsanity.com/paper/2605.08738/) (2026)
- [How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle](https://mlsanity.com/paper/2606.15716/) (2026)
- [AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning](https://mlsanity.com/paper/2603.18492/) (2026)
- [Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression](https://mlsanity.com/paper/2603.02217/) (2026)
- [Model Compression with Exact Budget Constraints via Riemannian Manifolds](https://mlsanity.com/paper/2605.00649/) (2026)
- [Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control](https://mlsanity.com/paper/2605.07182/) (2026)
- [Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts](https://mlsanity.com/paper/2605.28042/) (2026)
- [SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs](https://mlsanity.com/paper/2606.09886/) (2026)
- [Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding](https://mlsanity.com/paper/2607.16721/) (2026)
- [It Takes a MAESTRO To Prune Bad Experts](https://mlsanity.com/paper/2607.08601/) (2026)
- [ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration](https://mlsanity.com/paper/2608.24938/) (2026)
- [MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning](https://mlsanity.com/paper/2603.24044/) (2026)
- [HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts](https://mlsanity.com/paper/2605.13997/) (2026)
- [Pruning and Distilling Mixture-of-Experts into Dense Language Models](https://mlsanity.com/paper/2605.28207/) (2026)
- [ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression](https://mlsanity.com/paper/2605.29350/) (2026)
- [On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain](https://mlsanity.com/paper/2607.01444/) (2026)
- [Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models](https://mlsanity.com/paper/2608.04454/) (2026)
- [Higher-order pruning of experts in mixture-of-experts language models](https://mlsanity.com/paper/2609.18916/) (2026)
- [Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM](https://mlsanity.com/paper/2601.14327/) (2026)
- [UMoE:Unlocking Every Expert in Domain-Specific Training](https://mlsanity.com/paper/2607.11444/) (2026)
- [When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning](https://mlsanity.com/paper/2608.02940/) (2026)
- [Shape Mutating Expert Compression:LorExperts and BTExperts](https://mlsanity.com/paper/2608.07814/) (2026)
- [Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models](https://mlsanity.com/paper/2608.07890/) (2026)
- [Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs](https://mlsanity.com/paper/2608.21693/) (2026)
- [When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models](https://mlsanity.com/paper/2609.04453/) (2026)
- [Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models](https://mlsanity.com/paper/2609.04575/) (2026)

## Abstract (arXiv)

Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.