mlsanityData to 7 Oct 2026

Revisiting Single-gated Mixtures of Experts

arXiv 2304.05497 ↗First posted 11 Apr 2023cs.CV cs.LGcited by 10

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture of Experts (MoE) are rising in popularity as a means to train extremely large-scale models, yet allowing for a reasonable computational cost at inference time. Recent state-of-the-art approaches usually assume a large number of experts, and require training all experts jointly, which often lead to training instabilities such as the router collapsing In contrast, in this work, we propose to revisit the simple single-gate MoE, which allows for more practical training. Key to our work are (i) a base model branch acting both as an early-exit and an ensembling regularization scheme, (ii) a simple and efficient asynchronous training pipeline without router collapse issues, and finally (iii) a per-sample clustering-based initialization. We show experimentally that the proposed model obtains efficiency-to-accuracy trade-offs comparable with other more complex MoE, and outperforms non-mixture baselines. This showcases the merits of even a simple single-gate MoE, and motivates further exploration in this area.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 33 of its 56 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

33 of 56 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2019Searching for MobileNetV3cited by 1,984
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  12. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
21 more
  1. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  2. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  3. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  4. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  5. 2020Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixupcited by 260
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  7. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  8. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  9. 2020SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularizationcited by 149
  10. 2019Making Better Mistakes: Leveraging Class Hierarchies with Deep Networkscited by 112
  11. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  12. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  13. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 49
  14. 2020Scalable Transfer Learning with Expert Modelscited by 48
  15. 2017Deciding How to Decide: Dynamic Routing in Artificial Neural Networkscited by 33
  16. 2020Learning to Stop While Learning to Predictcited by 30
  17. 2021Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixupcited by 22
  18. 2017ME R-CNN: Multi-Expert R-CNN for Object Detectioncited by 13
  19. 2021MMF: Multi-Task Multi-Structure Fusion for Hierarchical Image Classificationcited by 5
  20. 2021Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPscited by 5
  21. 2017BranchConnect: Large-Scale Visual Recognition with Learned Branch Connectionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 2.

10 papers.

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 6
  3. 2025Semi-MoE: Mixture-of-Experts meets Semi-Supervised Histopathology Segmentationcited by 4
  4. 2025Mol-MoE: Training Preference-Guided Routers for Molecule Generationcited by 3
  5. 2026EMoE: Eigenbasis-Guided Routing for Mixture-of-Expertscited by 2
  6. 2024Enhancing Fast Feed Forward Networks with Load Balancing and a Master Leaf Nodecited by 1
  7. 2025CoCoAFusE: Beyond Mixtures of Experts via Model Fusioncited by 1
  8. 2026PACT: Peak-Aware Cross-Attention Graph Transformers for Efficient Storm-Surge Emulationcited by 1
  9. 2024GUMBEL-NERF: Representing Unseen Objects as Part-Compositional Neural Radiance Fieldscited by 0
  10. 2025Atmos-Bench: 3D Atmospheric Structures for Climate Insightcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  2. 2026∞-MoE: Generalizing Mixture of Experts to Infinite Experts
  3. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  4. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  5. 2026Towards a Statistical Understanding of Mixture-of-Experts
  6. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models
  7. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast
  8. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling