mlsanityData to 7 Oct 2026

Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping

arXiv 2404.19429 ↗First posted 30 Apr 2024cs.DC cs.LGcited by 22

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture-of-Expert (MoE) technique plays a crucial role in expanding the size of DNN model parameters. However, it faces the challenge of extended all-to-all communication latency during the training process. Existing methods attempt to mitigate this issue by overlapping all-to-all with expert computation. Yet, these methods frequently fall short of achieving sufficient overlap, consequently restricting the potential for performance enhancements. In our study, we extend the scope of this challenge by considering overlap at the broader training graph level. During the forward pass, we enable non-MoE computations to overlap with all-to-all through careful partitioning and pipelining. In the backward pass, we achieve overlap with all-to-all by scheduling gradient weight computations. We implement these techniques in Lancet, a system using compiler-based optimization to automatically enhance MoE model training. Our extensive evaluation reveals that Lancet significantly reduces the time devoted to non-overlapping communication, by as much as 77%. Moreover, it achieves a notable end-to-end speedup of up to 1.3 times when compared to the state-of-the-art solutions.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 24 of its 39 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

24 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  2. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  4. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2024Mixtral of Expertscited by 1,785
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
7 more
  1. 2021Hash Layers For Large Sparse Modelscited by 236
  2. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  3. 2018Glow: Graph Lowering Compiler Techniques for Neural Networkscited by 73
  4. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  5. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  6. 2019Priority-based Parameter Propagation for Distributed DNN Trainingcited by 21
  7. 2023Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelismcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 10.

22 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routingcited by 26
  3. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 26
  4. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  5. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 13
  6. 2025TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inferencecited by 11
  7. 2024Lazarus: Resilient and Elastic Training of Mixture-of-Experts Modelscited by 9
  8. 2025Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reorderingcited by 9
  9. 2024Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlappingcited by 8
  10. 2025SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decouplingcited by 5
  11. 2026LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Trainingcited by 5
  12. 2025X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platformscited by 3
10 more
  1. 2025FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Trainingcited by 2
  2. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 1
  3. 2026MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systemscited by 0
  4. 2026EMO: Frustratingly Easy Progressive Training of Extendable MoEcited by 0
  5. 2026HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUscited by 0
  6. 2026Resource-aware Computation-Communication Overlap for multi-GPU ML Workloadscited by 0
  7. 2026CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selectioncited by 0
  8. 2026UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpodscited by 0
  9. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 0
  10. 2026Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlapcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026∞-MoE: Generalizing Mixture of Experts to Infinite Experts
  2. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
  3. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  4. 2023Revisiting Single-gated Mixtures of Experts
  5. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  6. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  7. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
  8. 2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration