mlsanityData to 7 Oct 2026

ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development

arXiv 2407.20143 ↗First posted 29 Jul 2024cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Checkpointing to preserve training states is crucial during the development of Large Foundation Models (LFMs), for training resumption upon various failures or changes in GPU resources and parallelism configurations. In addition, saved checkpoints are dispatched to evaluation tasks or transferred across different training stages (e.g., from pre-training to post-training). All these scenarios require resharding distributed checkpoints from one parallelism to another. In production environments, different LFMs are trained with various frameworks and storage backends, depending on model sizes and training scales. A high-performance checkpointing system is needed to enable efficient checkpoint management at scale throughout the lifecycle of LFM development. We introduce ByteCheckpoint, an industrial-grade checkpointing system for large-scale LFM training. ByteCheckpoint features: a parallelism-agnostic checkpoint representation that enables efficient load-time checkpoint resharding; a generic checkpoint saving/loading workflow to accommodate multiple training frameworks and support different storage backends; full-stack optimizations to ensure high I/O efficiency and scalability; a suite of monitoring tools to streamline large-scale performance analysis and bottleneck detection. Compared to existing open-source checkpointing systems [52, 58], ByteCheckpoint significantly reduces runtime checkpoint stalls, achieving an average reduction of 54.20x. For saving and loading times, ByteCheckpoint achieves improvements of up to 9.96x and 8.80x, respectively.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 33 of its 71 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

33 of 71 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2016Layer Normalizationcited by 5,579
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
21 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  9. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  10. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  11. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  12. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  13. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  14. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  15. 2022Aryl: An Elastic Cluster Scheduler for Deep Learningcited by 27
  16. 2023Unicron: Economizing Self-Healing LLM Training at Scalecited by 16
  17. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 16
  18. 2024DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Modelscited by 16
  19. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 15
  20. 2023SWIFT: Expedited Failure Recovery for Large-scale DNN Trainingcited by 5
  21. 2024QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devicescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 13.

23 papers.

Sort:

By area:

  1. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  2. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20
  3. 2025Robust LLM Training Infrastructure at ByteDancecited by 14
  4. 2025Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Trainingcited by 6
  5. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  6. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  7. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Servingcited by 3
  8. 2025FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMscited by 2
  9. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  10. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 2
  11. 2024TrainMover: An Interruption-Resilient Runtime for ML Trainingcited by 1
  12. 2026LUMEN: Coordinated Failure Recovery for Distributed LLM Servingcited by 1
11 more
  1. 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
  2. 2025BootSeer: Analyzing and Mitigating Initialization Bottlenecks in Large-Scale LLM Trainingcited by 0
  3. 2026TStore: Rethinking AI Model Hub with Tensor-Centric Compressioncited by 0
  4. 2026From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUscited by 0
  5. 2026TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Trainingcited by 0
  6. 2026Orbax: Distributed Checkpointing with JAXcited by 0
  7. 2026PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpointcited by 0
  8. 2026The Life of a Token: from Words to Bits on the Wirecited by 0
  9. 2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0
  10. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 0
  11. 2026Leto: Fast In-Place Recovery for LLM Training on Surviving Hardwarecited by 0