mlsanityData to 7 Oct 2026

RAZOR: Pruning Replaceable Experts in LLMs

arXiv 2609.30465 ↗First posted 24 Sep 2026cs.LG cs.CLcited by 0

On the map: Mixture-of-experts → Expert routing strategies also fits: MoE large language models

Abstract (arXiv)

Mixture-of-experts (MoE) models activate only a few experts per token but store the entire expert pool. Pruning this pool requires identifying experts whose removal preserves model behavior. Routing frequency and output magnitude do not fully describe deletion damage, which also depends on how the surviving and replacement experts compensate for the removed output. We introduce RAZOR, a training-free pruning method based on consensus residuals, the deviations of expert outputs from their original weighted mixture. At a fixed layer input, these residuals give the exact output change for a single deletion under survivor renormalization and router refill. RAZOR aggregates this damage by conditional root mean square and selects experts under a layerwise budget using forward computation alone, without gradients, subset search, or recovery training. Against frequency, activation-norm, and REAP baselines on GLM-4.7-Flash and Qwen3.6-35B-A3B at 25% and 50% expert removal, it attains the highest macro average over nine reasoning-intensive tasks in all four model-budget settings, gaining 2.12-5.59 points over REAP and lowering reverse KL in all four. On DeepSeek-V4-Flash-0731 and Hy3, it also achieves the highest macro average among the three residual criteria. Local exactness does not guarantee better joint pruning. Generation analyses show changes in diversity, formatting, and termination despite higher task scores.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 36 of its 53 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

36 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026It Takes a MAESTRO To Prune Bad Experts9 Jul
  2. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs3 Jun
  3. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression10 Feb
  4. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy2 Oct
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 31

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  12. 2025Generalizing Verifiable Instruction Followingcited by 178
19 more
  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  2. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  3. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  4. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  5. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  6. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  7. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  8. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 31
  9. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  10. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  11. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
  12. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  13. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  14. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  15. 2025HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Spacecited by 3
  16. 2026Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Codingcited by 3
  17. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?cited by 2
  18. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Modelscited by 2
  19. 2026ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compressioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
  2. 2024SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
  3. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
  4. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
  5. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  7. 2026SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving
  8. 2026Higher-order pruning of experts in mixture-of-experts language models