# A Study of Checkpointing in Large Scale Training of Deep Neural Networks

arXiv: https://arxiv.org/abs/2012.00825 · HTML: https://mlsanity.com/paper/2012.00825/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2020.

## Built on by (most cited first, 11 of 11)

- [SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining](https://mlsanity.com/paper/2406.02214/) (2024)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editing](https://mlsanity.com/paper/2406.09404/) (2024)
- [FastPersist: Accelerating Model Checkpointing in Deep Learning](https://mlsanity.com/paper/2406.13768/) (2024)
- [I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey](https://mlsanity.com/paper/2404.10386/) (2024)
- [LiveTune: Dynamic Parameter Tuning for Feedback-Driven Optimization](https://mlsanity.com/paper/2311.17279/) (2023)
- [Rina: Enhancing Ring-AllReduce with In-network Aggregation in Distributed Model Training](https://mlsanity.com/paper/2407.19721/) (2024)
- [Ontology-Aligned Embeddings for Data-Driven Labour Market Analytics](https://mlsanity.com/paper/2509.04942/) (2025)
- [Intent-Driven Storage Systems: From Low-Level Tuning to High-Level Understanding](https://mlsanity.com/paper/2510.15917/) (2025)
- [A Multi-Component AI Framework for Computational Psychology: From Robust Predictive Modeling to Deployed Generative Dialogue](https://mlsanity.com/paper/2510.21720/) (2025)
- [Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation](https://mlsanity.com/paper/2602.14701/) (2026)

## Abstract (arXiv)

Deep learning (DL) applications are increasingly being deployed on HPC systems, to leverage the massive parallelism and computing power of those systems for DL model training. While significant effort has been put to facilitate distributed training by DL frameworks, fault tolerance has been largely ignored. In this work, we evaluate checkpoint-restart, a common fault tolerance technique in HPC workloads. We perform experiments with three state-of-the-art DL frameworks common in HPC Chainer, PyTorch, and TensorFlow). We evaluate the computational cost of checkpointing, file formats and file sizes, the impact of scale, and deterministic checkpointing. Our evaluation shows some critical differences in checkpoint mechanisms and exposes several bottlenecks in existing checkpointing implementations. We provide discussion points that can aid users in selecting a fault-tolerant framework to use in HPC. We also provide takeaway points that framework developers can use to facilitate better checkpointing of DL workloads in HPC.