# High Performance I/O For Large Scale Deep Learning

arXiv: https://arxiv.org/abs/2001.01858 · HTML: https://mlsanity.com/paper/2001.01858/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2020.

## Built on by (most cited first, 26 of 26)

- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [FFCV: Accelerating Training by Removing Data Bottlenecks](https://mlsanity.com/paper/2306.12517/) (2023)
- [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2203.15455/) (2022)
- [Deep Lake: a Lakehouse for Deep Learning](https://mlsanity.com/paper/2209.10785/) (2022)
- [Profiling and Improving the PyTorch Dataloader for high-latency Storage: A Technical Report](https://mlsanity.com/paper/2211.04908/) (2022)
- [How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study](https://mlsanity.com/paper/2306.03163/) (2023)
- [G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systems](https://mlsanity.com/paper/2401.04338/) (2024)
- [GlobalGeoTree: A Multi-Granular Vision-Language Dataset for Global Tree Species Classification](https://mlsanity.com/paper/2505.12513/) (2025)
- [High Throughput Training of Deep Surrogates from Large Ensemble Runs](https://mlsanity.com/paper/2309.16743/) (2023)
- [I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey](https://mlsanity.com/paper/2404.10386/) (2024)
- [123D: Unifying Multi-Modal Autonomous Driving Data at Scale](https://mlsanity.com/paper/2605.08084/) (2026)
- [FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching](https://mlsanity.com/paper/2604.06757/) (2026)
- [scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics](https://mlsanity.com/paper/2506.01883/) (2025)
- [Same model, better performance: the impact of shuffling on DNA Language Models benchmarking](https://mlsanity.com/paper/2510.12617/) (2025)
- [SignDATA: Data Pipeline for Sign Language Translation](https://mlsanity.com/paper/2604.20357/) (2026)
- [Differentiable programming for functional connectomics](https://mlsanity.com/paper/2206.00649/) (2022)
- [Detecting Network-based Internet Censorship via Latent Feature Representation Learning](https://mlsanity.com/paper/2209.05152/) (2022)
- [Corgi^2: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGD](https://mlsanity.com/paper/2309.01640/) (2023)
- [Multi-model learning by sequential reading of untrimmed videos for action recognition](https://mlsanity.com/paper/2401.14675/) (2024)
- [Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data](https://mlsanity.com/paper/2506.06235/) (2025)
- [GetBatch: Distributed Multi-Object Retrieval for ML Data Loading](https://mlsanity.com/paper/2602.22434/) (2026)
- [BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training](https://mlsanity.com/paper/2605.09994/) (2026)
- [Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets](https://mlsanity.com/paper/2606.29975/) (2026)
- [LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data](https://mlsanity.com/paper/2607.28880/) (2026)
- [FoldPipe: Bounded Remote Streaming of Native Molecular Shards with Asynchronous Prefetch](https://mlsanity.com/paper/2608.27029/) (2026)
- [Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines](https://mlsanity.com/paper/2610.03087/) (2026)

## Abstract (arXiv)

Training deep learning (DL) models on petascale datasets is essential for achieving competitive and state-of-the-art performance in applications such as speech, video analytics, and object recognition. However, existing distributed filesystems were not developed for the access patterns and usability requirements of DL jobs. In this paper, we describe AIStore, a highly scalable, easy-to-deploy storage system, and WebDataset, a standards-based storage format and library that permits efficient access to very large datasets. We compare system performance experimentally using image classification workloads and storing training data on a variety of backends, including local SSDs, single-node NFS, and two identical bare-metal clusters: HDFS and AIStore.