mlsanityData to 7 Oct 2026

CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition

arXiv 2402.02526 ↗First posted 4 Feb 2024cs.LGcited by 23

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, effective training of SMoE has proven to be challenging due to the representation collapse issue, which causes parameter redundancy and limited representation potentials. In this work, we propose a competition mechanism to address this fundamental challenge of representation collapse. By routing inputs only to experts with the highest neural response, we show that, under mild assumptions, competition enjoys the same convergence rate as the optimal estimator. We further propose CompeteSMoE, an effective and efficient algorithm to train large language models by deploying a simple router that predicts the competition outcomes. Consequently, CompeteSMoE enjoys strong performance gains from the competition routing policy while having low computation overheads. Our extensive empirical evaluations on two transformer architectures and a wide range of tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE compared to state-of-the-art SMoE strategies.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 40 of its 78 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

40 of 78 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts12 Dec
  2. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers2 Mar
  3. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  4. 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
  5. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  6. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2016HyperNetworkscited by 1,003
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
21 more
  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  3. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  5. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  7. 2019Recurrent Independent Mechanismscited by 219
  8. 2013Maxout Networkscited by 186
  9. 2013Deep Learning of Representations: Looking Forwardcited by 84
  10. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  11. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  12. 2019Convergence Rates for Gaussian Mixtures of Expertscited by 39
  13. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  14. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  15. 2021Transformers with Competitive Ensembles of Independent Mechanismscited by 19
  16. 2016A Universal Approximation Theorem for Mixture of Experts Modelscited by 17
  17. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  18. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  19. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  20. 2022Sparse Mixers: Combining MoE and Mixing to build a more efficient BERTcited by 12
  21. 2021A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 6; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  3. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  4. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  5. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  6. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  7. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionscited by 16
  8. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  9. 2025Tight Clusters Make Specialized Expertscited by 8
  10. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  11. 2024Elliptical Attentioncited by 7
  12. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
11 more
  1. 2024Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Expertscited by 5
  2. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  3. 2025Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measurescited by 3
  4. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  5. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  6. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 2
  7. 2026Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activationscited by 2
  8. 2026Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Expertscited by 1
  9. 2024Active partitioning: inverting the paradigm of active learningcited by 0
  10. 2026Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertscited by 0
  11. 2026Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition
  2. 2024SimSMoE: Solving Representational Collapse via Similarity Measure
  3. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  4. 2025S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
  5. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  6. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
  7. 2026SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
  8. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models