mlsanityData to 7 Oct 2026

A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts

arXiv 2405.16646 ↗First posted 26 May 2024cs.LGcited by 21

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

The sparsely gated mixture of experts (MoE) architecture sends different inputs to different subnetworks, i.e., experts, through trainable routers. MoE reduces the training computation significantly for large models, but its deployment can be still memory or computation expensive for some downstream tasks. Model pruning is a popular approach to reduce inference computation, but its application in MoE architecture is largely unexplored. To the best of our knowledge, this paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy, while significantly reducing the model size and the computational requirements. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models such as VMoE and E3MoE finetuned on benchmark datasets such as CIFAR10, CIFAR100, and ImageNet.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 58 of its 69 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

58 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022On the Adversarial Robustness of Mixture of Experts19 Oct
  2. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
38 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  8. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  12. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  13. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  14. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  15. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  16. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  17. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  18. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  19. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  20. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  21. 2019Structured Pruning of Large Language Modelscited by 201
  22. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  23. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  24. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  25. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  26. 2020Learning Parities with Neural Networkscited by 77
  27. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  28. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  29. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  30. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  31. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  32. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  33. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  34. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  35. 2021SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruningcited by 30
  36. 2020Fast Learning of Graph Neural Networks with Guaranteed Generalizability: One-hidden-layer Casecited by 24
  37. 2022Generalization Guarantee of Training Graph Convolutional Networks with Graph Topology Samplingcited by 23
  38. 2018Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropycited by 22

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 10.

21 papers.

Sort:

By area:

  1. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  4. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  5. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  6. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  7. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  8. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  9. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?cited by 2
  10. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  11. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  12. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
9 more
  1. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  2. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0
  3. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillationcited by 0
  4. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 0
  5. 2026Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guaranteescited by 0
  6. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  7. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  8. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  9. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  2. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
  3. 2023Revisiting Single-gated Mixtures of Experts
  4. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  5. 2023From Sparse to Soft Mixtures of Experts
  6. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
  7. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  8. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning