mlsanityData to 14 Oct 2026

TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training

arXiv 2302.09915 ↗First posted 20 Feb 2023cs.LG cs.AIcited by 15

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparsely gated Mixture-of-Expert (MoE) has demonstrated its effectiveness in scaling up deep neural networks to an extreme scale. Despite that numerous efforts have been made to improve the performance of MoE from the model design or system optimization perspective, existing MoE dispatch patterns are still not able to fully exploit the underlying heterogeneous network environments. In this paper, we propose TA-MoE, a topology-aware routing strategy for large-scale MoE trainging, from a model-system co-design perspective, which can dynamically adjust the MoE dispatch pattern according to the network topology. Based on communication modeling, we abstract the dispatch problem into an optimization objective and obtain the approximate dispatch pattern under different topologies. On top of that, we design a topology-aware auxiliary loss, which can adaptively route the data to fit in the underlying topology without sacrificing the model accuracy. Experiments show that TA-MoE can substantially outperform its counterparts on various hardware and model configurations, with roughly 1.01x-1.61x, 1.01x-4.77x, 1.25x-1.54x improvements over the popular DeepSpeed-MoE, FastMoE and FasterMoE.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 20 of its 29 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

20 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  10. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  12. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 334
6 more
  1. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  2. 2021Hash Layers For Large Sparse Modelscited by 237
  3. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  4. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  5. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training Systemcited by 36
  6. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 5; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  3. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  4. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  5. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  6. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  7. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  8. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Schedulingcited by 9
  9. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  10. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Schedulingcited by 7
  11. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  12. 2023Task-Based MoE for Multitask Multilingual Machine Translationcited by 5
3 more
  1. 2025Fine-grained MoE Load Balancing with Linear Programmingcited by 1
  2. 2025UrbanMind: Towards Urban General Intelligence via Tool-Enhanced Retrieval-Augmented Generation and Multilevel Optimizationcited by 0
  3. 2026Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Tutel: Adaptive Mixture-of-Experts at Scale
  2. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routing
  3. 2026Memory-Efficient Expert Routing for Distributed MoE Training
  4. 2025Load Balancing Mixture of Experts with Similarity Preserving Routers
  5. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformers
  6. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
  7. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
  8. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications