# DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models

arXiv: https://arxiv.org/abs/2406.10707 · HTML: https://mlsanity.com/paper/2406.10707/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://mlsanity.com/paper/2006.15704/) (2020)
- [M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining](https://mlsanity.com/paper/2110.03888/) (2021)
- [Unicron: Economizing Self-Healing LLM Training at Scale](https://mlsanity.com/paper/2401.00134/) (2023)
- [DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies](https://mlsanity.com/paper/2310.04610/) (2023)
- [TRANSOM: An Efficient Fault-Tolerant System for Training LLMs](https://mlsanity.com/paper/2310.10046/) (2023)
- [Paxos vs Raft: Have we reached consensus on distributed consensus?](https://mlsanity.com/paper/2004.05074/) (2020)

## Built on by (most cited first, 16 of 16)

- [TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training](https://mlsanity.com/paper/2410.06511/) (2024)
- [ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development](https://mlsanity.com/paper/2407.20143/) (2024)
- [Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading](https://mlsanity.com/paper/2410.21316/) (2024)
- [EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants](https://mlsanity.com/paper/2502.20309/) (2025)
- [MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training](https://mlsanity.com/paper/2408.04307/) (2024)
- [Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs](https://mlsanity.com/paper/2503.11901/) (2025)
- [Sparse Checkpointing for Fast and Reliable MoE Training](https://mlsanity.com/paper/2412.15411/) (2024)
- [FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs](https://mlsanity.com/paper/2509.03047/) (2025)
- [Harmonia: Enhancing Data Placement and Migration in Hybrid Storage Systems via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.20507/) (2025)
- [DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providers](https://mlsanity.com/paper/2601.16956/) (2026)
- [Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training](https://mlsanity.com/paper/2505.12815/) (2025)
- [MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall](https://mlsanity.com/paper/2509.02480/) (2025)
- [CCL-D: A High-Precision Diagnostic System for Slow and Hang Anomalies in Large-Scale Model Training](https://mlsanity.com/paper/2605.04478/) (2026)
- [ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload](https://mlsanity.com/paper/2605.11215/) (2026)
- [Twelve quick tips for designing AI-driven HPC workflows](https://mlsanity.com/paper/2606.07491/) (2026)
- [PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint](https://mlsanity.com/paper/2607.01646/) (2026)

## Abstract (arXiv)

LLMs have seen rapid adoption in all domains. They need to be trained on high-end high-performance computing (HPC) infrastructures and ingest massive amounts of input data. Unsurprisingly, at such a large scale, unexpected events (e.g., failures of components, instability of the software, undesirable learning patterns, etc.), are frequent and typically impact the training in a negative fashion. Thus, LLMs need to be checkpointed frequently so that they can be rolled back to a stable state and subsequently fine-tuned. However, given the large sizes of LLMs, a straightforward checkpointing solution that directly writes the model parameters and optimizer state to persistent storage (e.g., a parallel file system), incurs significant I/O overheads. To address this challenge, in this paper we study how to reduce the I/O overheads for enabling fast and scalable checkpointing for LLMs that can be applied at high frequency (up to the granularity of individual iterations) without significant impact on the training process. Specifically, we introduce a lazy asynchronous multi-level approach that takes advantage of the fact that the tensors making up the model and optimizer state shards remain immutable for extended periods of time, which makes it possible to copy their content in the background with minimal interference during the training process. We evaluate our approach at scales of up to 180 GPUs using different model sizes, parallelism settings, and checkpointing frequencies. The results show up to 48× faster checkpointing and 2.2× faster end-to-end training runtime compared with the state-of-art checkpointing approaches.