mlsanityData to 14 Oct 2026

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

arXiv 2510.13999 ↗First posted 15 Oct 2025cs.LG cs.AIcited by 44

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 25 of its 86 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

25 of 86 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts1 Mar
  3. 2024DeepSeek-V3 Technical Report27 Dec
  4. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  5. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts12 Jun
  6. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques4 Jun
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  8. 2024Mixtral of Experts8 Jan
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  6. 2025Kimi K2: Open Agentic Intelligencecited by 567
  7. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  8. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  9. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
  10. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  11. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  12. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
4 more
  1. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  2. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  3. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 20
  4. 2024The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapsecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 43.

44 papers.

Sort:

By area:

  1. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Expertscited by 14
  2. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  3. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  4. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  5. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  6. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  7. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  8. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  9. 2026Model Compression with Exact Budget Constraints via Riemannian Manifoldscited by 3
  10. 2026Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Controlcited by 3
  11. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  12. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
32 more
  1. 2026Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Codingcited by 3
  2. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  3. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Accelerationcited by 2
  4. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuningcited by 1
  5. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  6. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1
  7. 2026ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compressioncited by 1
  8. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  9. 2026Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Modelscited by 1
  10. 2026Higher-order pruning of experts in mixture-of-experts language modelscited by 1
  11. 2026Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLMcited by 0
  12. 2026UMoE:Unlocking Every Expert in Domain-Specific Trainingcited by 0
  13. 2026When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruningcited by 0
  14. 2026Shape Mutating Expert Compression:LorExperts and BTExpertscited by 0
  15. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  16. 2026Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMscited by 0
  17. 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Modelscited by 0
  18. 2026Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Modelscited by 0
  19. 2026ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMscited by 0
  20. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0
  21. 2026MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Modelscited by 0
  22. 2026What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexitycited by 0
  23. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0
  24. 2026Low-Bit Recurrent States in Hybrid Language Modelscited by 0
  25. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuitcited by 0
  26. 2026SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Servingcited by 0
  27. 2026BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decodingcited by 0
  28. 2026DIET: Deletion-response Expert Trimming for Video Diffusion Transformerscited by 0
  29. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Expertscited by 0
  30. 2026Dynamic Expert Pruning for Multi-Agent Systemscited by 0
  31. 2026PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compressioncited by 0
  32. 2026SLEM: Spectrum-Aware Latent Expert Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
  2. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
  3. 2026Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression
  4. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
  5. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
  6. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  7. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
  8. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition