I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey
Not on the map. Why?
Abstract (arXiv)
Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this paper, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 50 of its 163 references that are in the map. First posted on arXiv 16 Apr 2024.
Builds on
50 of 163 references.
References · 50
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
- 2016An overview of gradient descent optimization algorithmscited by 1,033
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
38 more
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
- 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
- 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
- 2018The UCR Time Series Archivecited by 431
- 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
- 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
- 2019MLPerf Inference Benchmarkcited by 199
- 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
- 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 152
- 2021Asynchronous Federated Learning on Heterogeneous Devices: A Surveycited by 93
- 2015Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmarkcited by 88
- 2020Fast WordPiece Tokenizationcited by 87
- 2021Federated Learning: Opportunities and Challengescited by 84
- 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
- 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
- 2021tf.data: A Machine Learning Data Processing Frameworkcited by 41
- 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
- 2017AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Trainingcited by 33
- 2020High Performance I/O For Large Scale Deep Learningcited by 26
- 2017Convergence Analysis of Distributed Stochastic Gradient Descent with Shufflingcited by 23
- 2017Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutescited by 22
- 2016Fathom: Reference Workloads for Modern Deep Learning Methodscited by 20
- 2022tf.data service: A Case for Disaggregating ML Input Data Processingcited by 16
- 2021MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systemscited by 15
- 2024Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real Systemcited by 15
- 2024Data Readiness for AI: A 360-Degree Surveycited by 15
- 2017Large-scale analysis of disease pathways in the human interactomecited by 14
- 2018CosmoFlow: Using Deep Learning to Learn the Universe at Scalecited by 14
- 2021Clairvoyant Prefetching for Distributed Machine Learning I/Ocited by 14
- 2022Deep Lake: a Lakehouse for Deep Learningcited by 13
- 2020A Study of Checkpointing in Large Scale Training of Deep Neural Networkscited by 11
- 2020On Efficient Constructions of Checkpointscited by 7
- 2022Profiling and Improving the PyTorch Dataloader for high-latency Storage: A Technical Reportcited by 5
- 2019Progressive Compressed Records: Taking a Byte out of Deep Learning Datacited by 4
- 2021Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspectivecited by 4
- 2020tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloadscited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.