mlsanityData to 7 Oct 2026

Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models

arXiv 2407.04656 ↗First posted 5 Jul 2024cs.DC cs.LGcited by 9

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Sparsely-activated Mixture-of-Experts (MoE) architecture has increasingly been adopted to further scale large language models (LLMs). However, frequent failures still pose significant challenges as training scales. The cost of even a single failure is significant, as all GPUs need to idle wait until the failure is resolved, potentially losing considerable training progress as training has to restart from checkpoints. This problem is exacerbated by the growing use of spot instances on public clouds for model training, which despite offering substantial cost savings, introduce frequent preemptions-essentially failures that regularly occur throughout the training process. Existing solutions for efficient fault-tolerant training either lack elasticity or rely on building resiliency into pipeline parallelism, which cannot be applied to MoE models due to the expert parallelism strategy adopted by the MoE architecture. We present Lazarus, a system for resilient and elastic training of MoE models. Lazarus adaptively allocates expert replicas to address the inherent imbalance in expert workload and speeds up training, while a provably optimal expert placement algorithm is developed to maximize the probability of recovery upon failures. Through adaptive expert placement and a flexible token dispatcher, Lazarus can also fully utilize all available nodes after failures, leaving no GPU idle. Our evaluation shows that Lazarus outperforms existing MoE training systems by up to 5.7x under frequent node failures and 3.4x on a real spot instance trace.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 14 of its 46 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

14 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping30 Apr
  2. 2022HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System28 Mar

Other references · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2024Mixtral of Expertscited by 1,785
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  9. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  10. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  11. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  12. 2023Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 3.

9 papers.

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 12
  3. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 10
  4. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Schedulingcited by 7
  5. 2025HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUscited by 7
  6. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  7. 2026Making MoE-based LLM Inference Resilient with Tarragoncited by 2
  8. 2026Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inferencecited by 1
  9. 2026From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sparse Checkpointing for Fast and Reliable MoE Training
  2. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  3. 2024MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
  4. 2024Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation
  5. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
  6. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  7. 2024Mixture of Diverse Size Experts
  8. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling