mlsanityData to 7 Oct 2026

MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts

arXiv 2410.14574 ↗First posted 18 Oct 2024cs.LG cs.AI cs.CLcited by 9

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Sparse Mixture of Experts (SMoE) has become the key to unlocking unparalleled scalability in deep learning. SMoE has the potential to exponentially increase parameter count while maintaining the efficiency of the model by only activating a small subset of these parameters for a given sample. However, it has been observed that SMoE suffers from unstable training and has difficulty adapting to new distributions, leading to the model's lack of robustness to data contamination. To overcome these limitations, we first establish a connection between the dynamics of the expert representations in SMoEs and gradient descent on a multi-objective optimization problem. Leveraging our framework, we then integrate momentum into SMoE and propose a new family of SMoEs named MomentumSMoE. We theoretically prove and numerically demonstrate that MomentumSMoE is more stable and robust than SMoE. In particular, we verify the advantages of MomentumSMoE over SMoE on a variety of practical tasks including ImageNet-1K object recognition and WikiText-103 language modeling. We demonstrate the applicability of MomentumSMoE to many types of SMoE models, including those in the Sparse MoE model for vision (V-MoE) and the Generalist Language Model (GLaM). We also show that other advanced momentum-based optimization methods, such as Adam, can be easily incorporated into the MomentumSMoE framework for designing new SMoE models with even better performance, almost negligible additional computation cost, and simple implementations.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 60 of its 78 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

60 of 78 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  13. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  14. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  15. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  16. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  17. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  18. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  19. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  20. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  21. 2023From Sparse to Soft Mixtures of Expertscited by 245
  22. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  23. 2021Partial success in closing the gap between human and machine visioncited by 185
  24. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  25. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  26. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  27. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  28. 2017A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortionscited by 107
  29. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  30. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 98
  31. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  32. 2018Negative Momentum for Improved Game Dynamicscited by 59
  33. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 54
  34. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  35. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  36. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 49
  37. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  38. 2023Brainformers: Trading Simplicity for Efficiencycited by 31

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2025Tight Clusters Make Specialized Expertscited by 8
  2. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  3. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
  4. 2025CAMEx: Curvature-aware Merging of Expertscited by 4
  5. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 2
  6. 2026MuonSSM: Orthogonalizing State Space Models for Sequence Modelingcited by 2
  7. 2025Mixture of Group Experts for Learning Invariant Representationscited by 1
  8. 2025Selective Sinkhorn Routing for Improved Sparse Mixture of Expertscited by 1
  9. 2026Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
  2. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  3. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
  4. 2025Sigma-MoE-Tiny Technical Report
  5. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity
  6. 2025Improving Routing in Sparse Mixture of Experts with Graph of Tokens
  7. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  8. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization