mlsanityData to 14 Oct 2026

It Takes a MAESTRO To Prune Bad Experts

arXiv 2607.08601 ↗First posted 9 Jul 2026cs.CLcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 52 of its 65 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

52 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy2 Oct
  2. 2023Soft Merging of Experts with Adaptive Routing6 Jun

Other references · 50

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
38 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  9. 2018CoQA: A Conversational Question Answering Challengecited by 683
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  13. 2018Gender Bias in Coreference Resolutioncited by 442
  14. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  15. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  16. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  17. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  18. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  19. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  20. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  21. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  22. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 59
  23. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  24. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  25. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  26. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 37
  27. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 36
  28. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 35
  29. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  30. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 27
  31. 2023Fairness-Aware Structured Pruning in Transformerscited by 22
  32. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  33. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  34. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 17
  35. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 13
  36. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Modelscited by 11
  37. 2024Large Language Models Are Overparameterized Text Encoderscited by 6
  38. 2026ActTail: Global Activation Sparsity in Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Codingcited by 3
  2. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoE Pathfinder: Trajectory-driven Expert Pruning
  2. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
  3. 2026Higher-order pruning of experts in mixture-of-experts language models
  4. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  5. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  6. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
  7. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
  8. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs