mlsanityData to 7 Oct 2026

High Performance I/O For Large Scale Deep Learning

arXiv 2001.01858 ↗First posted 7 Jan 2020cs.DCcited by 25

Not on the map. Why?

Abstract (arXiv)

Training deep learning (DL) models on petascale datasets is essential for achieving competitive and state-of-the-art performance in applications such as speech, video analytics, and object recognition. However, existing distributed filesystems were not developed for the access patterns and usability requirements of DL jobs. In this paper, we describe AIStore, a highly scalable, easy-to-deploy storage system, and WebDataset, a standards-based storage format and library that permits efficient access to very large datasets. We compare system performance experimentally using image classification workloads and storing training data on a variety of backends, including local SSDs, single-node NFS, and two identical bare-metal clusters: HDFS and AIStore.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 4; 2024: 3; 2025: 4; 2026: 8.

25 papers.

Sort:

By area:

  1. 2021Advances in Neural Renderingcited by 365
  2. 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
  3. 2022WeNet 2.0: More Productive End-to-End Speech Recognition Toolkitcited by 46
  4. 2022Deep Lake: a Lakehouse for Deep Learningcited by 13
  5. 2022Profiling and Improving the PyTorch Dataloader for high-latency Storage: A Technical Reportcited by 5
  6. 2023How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Studycited by 4
  7. 2024G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systemscited by 4
  8. 2025GlobalGeoTree: A Multi-Granular Vision-Language Dataset for Global Tree Species Classificationcited by 4
  9. 2023High Throughput Training of Deep Surrogates from Large Ensemble Runscited by 3
  10. 2024I/O in Machine Learning Applications on HPC Systems: A 360-degree Surveycited by 3
  11. 2026123D: Unifying Multi-Modal Autonomous Driving Data at Scalecited by 2
  12. 2025scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omicscited by 1
13 more
  1. 2025Same model, better performance: the impact of shuffling on DNA Language Models benchmarkingcited by 1
  2. 2026FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matchingcited by 1
  3. 2026SignDATA: Data Pipeline for Sign Language Translationcited by 1
  4. 2022Differentiable programming for functional connectomicscited by 0
  5. 2022Detecting Network-based Internet Censorship via Latent Feature Representation Learningcited by 0
  6. 2023Corgi^2: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGDcited by 0
  7. 2024Multi-model learning by sequential reading of untrimmed videos for action recognitioncited by 0
  8. 2025Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Datacited by 0
  9. 2026GetBatch: Distributed Multi-Object Retrieval for ML Data Loadingcited by 0
  10. 2026BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Trainingcited by 0
  11. 2026Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasetscited by 0
  12. 2026LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Datacited by 0
  13. 2026FoldPipe: Bounded Remote Streaming of Native Molecular Shards with Asynchronous Prefetchcited by 0