mlsanityData to 7 Oct 2026

STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning

arXiv 2409.06211 ↗First posted 10 Sep 2024cs.LG cs.CLcited by 30

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-experts (MoEs) have been adopted for reducing inference costs by sparsely activating experts in Large language models (LLMs). Despite this reduction, the massive number of experts in MoEs still makes them expensive to serve. In this paper, we study how to address this, by pruning MoEs. Among pruning methodologies, unstructured pruning has been known to achieve the highest performance for a given pruning ratio, compared to structured pruning, since the latter imposes constraints on the sparsification structure. This is intuitive, as the solution space of unstructured pruning subsumes that of structured pruning. However, our counterintuitive finding reveals that expert pruning, a form of structured pruning, can actually precede unstructured pruning to outperform unstructured-only pruning. As existing expert pruning, requiring O(frackn√n) forward passes for n experts, cannot scale for recent MoEs, we propose a scalable alternative with O(1) complexity, yet outperforming the more expensive methods. The key idea is leveraging a latent structure between experts, based on behavior similarity, such that the greedy decision of whether to prune closely captures the joint pruning effect. Ours is highly effective -- for Snowflake Arctic, a 480B-sized MoE with 128 experts, our method needs only one H100 and two hours to achieve nearly no loss in performance with 40% sparsity, even in generative tasks such as GSM8K, where state-of-the-art unstructured pruning fails to. The code will be made publicly available.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 42 of its 66 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

42 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  2. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  3. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  4. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  7. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep

Other references · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
23 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  7. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  8. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  9. 2021Accelerating Sparse Deep Neural Networkscited by 213
  10. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  11. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  12. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  13. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  14. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  15. 2019Learning Sparse Networks Using Targeted Dropoutcited by 40
  16. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  17. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  18. 2022Coarsening the Granularity: Towards Structurally Sparse Lottery Ticketscited by 27
  19. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  20. 2020Deep Transformers with Latent Depthcited by 17
  21. 2020Know What You Don't Need: Single-Shot Meta-Pruning for Attention Headscited by 16
  22. 2021Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modelingcited by 14
  23. 2021Adaptive Sparse Transformer for Multilingual Translationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 19.

30 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 26
  3. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  4. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  5. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  6. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  7. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  8. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  9. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 8
  10. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  11. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  12. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
18 more
  1. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  2. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  3. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  4. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?cited by 2
  5. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Modelscited by 2
  6. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  7. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  8. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inferencecited by 1
  9. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  10. 2026SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inferencecited by 1
  11. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  12. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
  13. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  14. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  15. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  16. 2026ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMscited by 0
  17. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0
  18. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Higher-order pruning of experts in mixture-of-experts language models
  2. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  3. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
  4. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  5. 2025Automatic Pruning Discovery for Large Language Models
  6. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
  7. 2025Pruning Laws for Large Language Models
  8. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models