mlsanityData to 14 Oct 2026

Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates

arXiv 2309.08125 ↗First posted 15 Sep 2023cs.DC cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Oobleck enables resilient distributed training of large DNN models with guaranteed fault tolerance. It takes a planning-execution co-design approach, where it first generates a set of heterogeneous pipeline templates and instantiates at least f+1 logically equivalent pipeline replicas to tolerate any f simultaneous failures. During execution, it relies on already-replicated model states across the replicas to provide fast recovery. Oobleck provably guarantees that some combination of the initially created pipeline templates can be used to cover all available resources after f or fewer simultaneous failures, thereby avoiding resource idling at all times. Evaluation on large DNN models with billions of parameters shows that Oobleck provides consistently high throughput, and it outperforms state-of-the-art fault tolerance solutions like Bamboo and Varuna by up to 29.6x.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 24 of its 54 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

24 of 54 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  10. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  11. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  12. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
12 more
  1. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  2. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  3. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  4. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  5. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  6. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  7. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  8. 2021Torch.fx: Practical Program Capture and Transformation for Deep Learning in Pythoncited by 40
  9. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  10. 2021Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameterscited by 17
  11. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  12. 2022SpotLake: Diverse Spot Instance Dataset Archive Servicecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 13; 2026: 13.

40 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  2. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  3. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  4. 2024ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Developmentcited by 24
  5. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  6. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 16
  7. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  8. 2025Robust LLM Training Infrastructure at ByteDancecited by 16
  9. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  10. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 11
  11. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 10
  12. 2025Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checkscited by 10
28 more
  1. 2024Multimodal Music Generation with Explicit Bridges and Retrieval Augmentationcited by 6
  2. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  3. 2026Training LLMs with Fault Tolerant HSDP on 100,000 GPUscited by 6
  4. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  5. 2026Decoupled DiLoCo for Resilient Distributed Pre-trainingcited by 4
  6. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  7. 2025OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Trainingcited by 3
  8. 2025TrainVerify: Equivalence-Based Verification for Distributed LLM Trainingcited by 3
  9. 2024Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Surveycited by 2
  10. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  11. 2026DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Trainingcited by 2
  12. 2023Exploring the Robustness of Decentralized Training for Large Language Modelscited by 1
  13. 2024TrainMover: An Interruption-Resilient Runtime for ML Trainingcited by 1
  14. 2025Orthogonal Calibration for Asynchronous Federated Learningcited by 1
  15. 2025All is Not Lost: LLM Recovery without Checkpointscited by 1
  16. 2026DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providerscited by 1
  17. 2026LUMEN: Coordinated Failure Recovery for Distributed LLM Servingcited by 1
  18. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 1
  19. 2025Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Trainingcited by 0
  20. 2025Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelizationcited by 0
  21. 2025Crash-Consistent Checkpointing for AI Training on macOS/APFScited by 0
  22. 2026From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUscited by 0
  23. 2026ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workloadcited by 0
  24. 2026PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpointcited by 0
  25. 2026When Is Availability-Aware Training Worth It? A Benchmark and Empirical Study of Interruption-Resilient Optimization Under Predictable Compute Schedulescited by 0
  26. 2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0
  27. 2026Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointingcited by 0
  28. 2026Leto: Fast In-Place Recovery for LLM Training on Surviving Hardwarecited by 0