mlsanityData to 7 Oct 2026

SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing

arXiv 2212.05191 ↗First posted 10 Dec 2022cs.LGcited by 3

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

The mixture of Expert (MoE) parallelism is a recent advancement that scales up the model size with constant computational cost. MoE selects different sets of parameters (i.e., experts) for each incoming token, resulting in a sparsely-activated model. Despite several successful applications of MoE, its training efficiency degrades significantly as the number of experts increases. The routing stage in MoE relies on the efficiency of the All2All communication collective, which suffers from network congestion and has poor scalability. To mitigate these issues, we introduce SMILE, which exploits heterogeneous network bandwidth and splits a single-step routing into bi-level routing. Our experimental results show that the proposed method obtains a 2.5x speedup over Switch Transformer in terms of pretraining throughput on the Colossal Clean Crawled Corpus without losing any convergence speed.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 15 of its 24 references that are in the map. First posted on arXiv 10 Dec 2022.

Builds on

15 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
2 more
  1. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  2. 2018MaskGAN: Better Text Generation via Filling in the______cited by 89

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 2.

3 papers.

  1. 2023Provable Pathways: Learning Multiple Tasks over Multiple Pathscited by 3
  2. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  3. 2025Stable-MoE: Lyapunov-based Token Routing for Distributed Mixture-of-Experts Training over Edge Networkscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
  2. 2026Routing-Free Mixture-of-Experts
  3. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  4. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
  5. 2026Memory-Efficient Expert Routing for Distributed MoE Training
  6. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routing
  7. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism
  8. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation