mlsanityData to 7 Oct 2026

SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts

arXiv 2404.05089 ↗First posted 7 Apr 2024cs.CL cs.LGcited by 40

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

The advancement of deep learning has led to the emergence of Mixture-of-Experts (MoEs) models, known for their dynamic allocation of computational resources based on input. Despite their promise, MoEs face challenges, particularly in terms of memory requirements. To address this, our work introduces SEER-MoE, a novel two-stage framework for reducing both the memory footprint and compute requirements of pre-trained MoE models. The first stage involves pruning the total number of experts using a heavy-hitters counting guidance, while the second stage employs a regularization-based fine-tuning strategy to recover accuracy loss and reduce the number of activated experts during inference. Our empirical studies demonstrate the effectiveness of our method, resulting in a sparse MoEs model optimized for inference efficiency with minimal accuracy trade-offs.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 8 of its 15 references that are in the map. First posted on arXiv 7 Apr 2024.

Builds on

8 of 15 references.

References · 8

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  7. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  8. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 18.

40 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  3. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  4. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  5. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  6. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 26
  7. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  8. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  9. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
  10. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  11. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  12. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
28 more
  1. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  2. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  3. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  4. 2024Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruningcited by 10
  5. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Modelscited by 10
  6. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  7. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  8. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  9. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  10. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Modelscited by 6
  11. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  12. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  13. 2026Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inferencecited by 3
  14. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  15. 2026Temporally Extended Mixture-of-Experts Modelscited by 2
  16. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Modelscited by 2
  17. 2026From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Modelscited by 2
  18. 2025PreMoE: Proactive Inference for Efficient Mixture-of-Expertscited by 1
  19. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  20. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0
  21. 2025FAIR-Pruner: A Flexible Framework for Automatic Layer-Wise Pruning via Tolerance of Differencecited by 0
  22. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Expertscited by 0
  23. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  24. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  25. 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMscited by 0
  26. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0
  27. 2026BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decodingcited by 0
  28. 2026Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  2. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
  3. 2026MoE Lens -- An Expert Is All You Need
  4. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
  5. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  7. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  8. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design