mlsanityData to 14 Oct 2026

FastPersist: Accelerating Model Checkpointing in Deep Learning

arXiv 2406.13768 ↗First posted 19 Jun 2024cs.DC cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we propose FastPersist to accelerate checkpoint creation in DL training. FastPersist combines three novel techniques: (i) NVMe optimizations for faster checkpoint writes to SSDs, (ii) efficient write parallelism using the available SSDs in training environments, and (iii) overlapping checkpointing with independent training computations. Our evaluation using real world dense and sparse DL models shows that FastPersist creates checkpoints in persistent storage up to 116x faster than baseline, and enables per-iteration checkpointing with negligible overhead.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 17 of its 47 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

17 of 47 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  11. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  12. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
5 more
  1. 2020Analyzing and Mitigating Data Stalls in DNN Trainingcited by 36
  2. 2021Clairvoyant Prefetching for Distributed Machine Learning I/Ocited by 14
  3. 2020Time-based Sequence Model for Personalization and Recommendation Systemscited by 11
  4. 2020A Study of Checkpointing in Large Scale Training of Deep Neural Networkscited by 11
  5. 2022Hardware/Software Co-Programmable Framework for Computational SSDs to Accelerate Deep Learning Service on Large-Scale Graphscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Paralleliscited by 9
  2. 2024MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Trainingcited by 4
  3. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  4. 2026DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providerscited by 1
  5. 2025Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Trainingcited by 0
  6. 2026PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpointcited by 0