mlsanityData to 14 Oct 2026

Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs

arXiv 2509.10377 ↗First posted 12 Sep 2025cs.CLcited by 25

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse Mixture-of-Experts (SMoE) architectures are widely used in large language models (LLMs) due to their computational efficiency. However, though only a few experts are activated for each token, SMoE still requires loading all expert parameters, leading to high memory usage and challenges in deployment. Previous work has tried to reduce the overhead by pruning and merging experts, but primarily focused on expert-level operations, leaving neuron-level structure underexplored. We propose DERN (Dropping Experts, Recombining Neurons), a task-agnostic and retraining-free framework for expert pruning and reconstruction. We observe that experts are often misaligned and contain semantic conflicts at the neuron level, which poses challenges for direct merging. To solve this, DERN works in three steps: it first prunes redundant experts using router statistics; then it decomposes them into neuron-level expert segments, assigning each segment to its most compatible retained expert; and finally, it merges segments within each retained expert to build a compact representation. Experiments on Mixtral, Qwen, and DeepSeek SMoE models show that DERN improves performance by more than 5% on commonsense reasoning and MMLU benchmarks under 50% expert sparsity, without extra training. It also greatly reduces the number of experts and memory usage, making SMoE LLMs easier to deploy in practice.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 28 of its 41 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

28 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Delta Decompression for MoE-based LLMs Compression24 Feb
  2. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning26 Nov
  3. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition1 Nov
  4. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  5. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More8 Oct
  6. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  12. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
9 more
  1. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  2. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  3. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 59
  4. 2024SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Modelscited by 25
  5. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 24
  6. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  7. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  8. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  9. 2025Enhancing Large Language Models through Structured Reasoningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 21.

25 papers.

Sort:

By area:

  1. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  2. 2025When TableQA Meets Noise: A Dual Denoising Framework for Complex Questions and Large-scale Tablescited by 10
  3. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  4. 2026XInsight: Integrative Stage-Consistent Psychological Counseling Support Agents for Digital Well-Beingcited by 9
  5. 2026Rethinking Table Pruning in TableQA: From Sequential Revisions to Gold Trajectory-Supervised Parallel Searchcited by 8
  6. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 8
  7. 2026FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Modelscited by 8
  8. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  9. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  10. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  11. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  12. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
13 more
  1. 2025Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuningcited by 1
  2. 2026Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Modelscited by 1
  3. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  4. 2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
  5. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  6. 2025Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignmentcited by 0
  7. 2026OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scalecited by 0
  8. 2026Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learningcited by 0
  9. 2026EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiationcited by 0
  10. 2026Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Modelscited by 0
  11. 2026Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generationcited by 0
  12. 2026VIA-SD: Verification via Intra-Model Routing for Speculative Decodingcited by 0
  13. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
  2. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  3. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
  4. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
  5. 2026NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension
  6. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
  7. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
  8. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models