mlsanityData to 7 Oct 2026

Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling

arXiv 2410.17043 ↗First posted 22 Oct 2024cs.LG cs.NIcited by 7

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

As machine learning models scale in size and complexity, their computational requirements become a significant barrier. Mixture-of-Experts (MoE) models alleviate this issue by selectively activating relevant experts. Despite this, MoE models are hindered by high communication overhead from all-to-all operations, low GPU utilization due to the synchronous communication constraint, and complications from heterogeneous GPU environments. This paper presents Aurora, which optimizes both model deployment and all-to-all communication scheduling to address these challenges in MoE inference. Aurora achieves minimal communication times by strategically ordering token transmissions in all-to-all communications. It improves GPU utilization by colocating experts from different models on the same device, avoiding the limitations of synchronous all-to-all communication. We analyze Aurora's optimization strategies theoretically across four common GPU cluster settings: exclusive vs. colocated models on GPUs, and homogeneous vs. heterogeneous GPUs. Aurora provides optimal solutions for three cases, and for the remaining NP-hard scenario, it offers a polynomial-time sub-optimal solution with only a 1.07x degradation from the optimal. Aurora is the first approach to minimize MoE inference time via optimal model deployment and communication scheduling across various scenarios. Evaluations demonstrate that Aurora significantly accelerates inference, achieving speedups of up to 2.38x in homogeneous clusters and 3.54x in heterogeneous environments. Moreover, Aurora enhances GPU utilization by up to 1.5x compared to existing methods.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 21 of its 43 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

21 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models5 Jul
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  3. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  5. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  6. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan
  8. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 13

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  8. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  9. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  10. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  11. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  12. 2023TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Trainingcited by 15
1 more
  1. 2024HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesiscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  3. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  4. 2026SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networkscited by 3
  5. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scalingcited by 1
  6. 2025RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Trainingcited by 1
  7. 2025Puzzle: Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
  2. 2026Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
  3. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  4. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  5. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  6. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  7. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  8. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models