# Accelerating Data Loading in Deep Neural Network Training

arXiv: https://arxiv.org/abs/1910.01196 · HTML: https://mlsanity.com/paper/1910.01196/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 10 of 10)

- [Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training](https://mlsanity.com/paper/2108.09373/) (2021)
- [Clairvoyant Prefetching for Distributed Machine Learning I/O](https://mlsanity.com/paper/2101.08734/) (2021)
- [Profiling and Improving the PyTorch Dataloader for high-latency Storage: A Technical Report](https://mlsanity.com/paper/2211.04908/) (2022)
- [NeuGuard: Lightweight Neuron-Guided Defense against Membership Inference Attacks](https://mlsanity.com/paper/2206.05565/) (2022)
- [Graph Learning at Scale: Characterizing and Optimizing Pre-Propagation GNNs](https://mlsanity.com/paper/2504.13266/) (2025)
- [SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates](https://mlsanity.com/paper/2211.00224/) (2022)
- [Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks](https://mlsanity.com/paper/2304.08925/) (2023)
- [Learning Single-Image Super-Resolution in the JPEG Compressed Domain](https://mlsanity.com/paper/2512.04284/) (2025)
- [Joint Optimization of Storage and Loading for High-Performance 3D Point Cloud Data Processing](https://mlsanity.com/paper/2603.16945/) (2026)
- [Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines](https://mlsanity.com/paper/2610.03087/) (2026)

## Abstract (arXiv)

Data loading can dominate deep neural network training time on large-scale systems. We present a comprehensive study on accelerating data loading performance in large-scale distributed training. We first identify performance and scalability issues in current data loading implementations. We then propose optimizations that utilize CPU resources to the data loader design. We use an analytical model to characterize the impact of data loading on the overall training time and establish the performance trend as we scale up distributed training. Our model suggests that I/O rate limits the scalability of distributed training, which inspires us to design a locality-aware data loading method. By utilizing software caches, our method can drastically reduce the data loading communication volume in comparison with the original data loading implementation. Finally, we evaluate the proposed optimizations with various experiments. We achieved more than 30x speedup in data loading using 256 nodes with 1,024 learners.