mlsanityData to 13 Oct 2026

Clairvoyant Prefetching for Distributed Machine Learning I/O

arXiv 2101.08734 ↗First posted 21 Jan 2021cs.DC cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

I/O is emerging as a major bottleneck for machine learning training, especially in distributed environments. Indeed, at large scale, I/O takes as much as 85% of training time. Addressing this I/O bottleneck necessitates careful optimization, as optimal data ingestion pipelines differ between systems, and require a delicate balance between access to local storage, external filesystems, and remote nodes. We introduce NoPFS, a machine learning I/O middleware, which provides a scalable, flexible, and easy-to-use solution to the I/O bottleneck. NoPFS uses clairvoyance: Given the seed generating the random access pattern for training with SGD, it can exactly predict when and where a sample will be accessed. We combine this with an analysis of access patterns and a performance model to provide distributed caching policies that adapt to different datasets and storage hierarchies. NoPFS reduces I/O times and improves end-to-end training by up to 5.4x on the ImageNet-1k, ImageNet-22k, and CosmoFlow datasets.

Key facts. Cited by 14 papers in the map (data through 13 Oct 2026). Builds on 26 of its 92 references that are in the map. First posted on arXiv 21 Jan 2021.

Builds on

26 of 92 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,488
  5. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,248
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,142
  7. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,058
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  9. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 617
  10. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  11. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  12. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
14 more
  1. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 277
  2. 2014cuDNN: Efficient Primitives for Deep Learningcited by 243
  3. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  4. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  5. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 79
  6. 2015Why Random Reshuffling Beats Stochastic Gradient Descentcited by 78
  7. 2018Exascale Deep Learning for Climate Analyticscited by 28
  8. 2018CosmoFlow: Using Deep Learning to Learn the Universe at Scalecited by 14
  9. 2020The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelismcited by 11
  10. 2019Accelerating Data Loading in Deep Neural Network Trainingcited by 10
  11. 2018Characterizing Deep-Learning I/O Workloads in TensorFlowcited by 5
  12. 2018Detecting Work Zones in SHRP 2 NDS Videos Using Deep Learning Based Computer Visioncited by 3
  13. 2019Parallelizing Training of Deep Generative Models on Massive Scientific Datasetscited by 3
  14. 2020Beyond Worst-case Analysis of Multicore Caching Strategiescited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 3; 2025: 2; 2026: 1.

14 papers.

Sort:

  1. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  2. 2022Compressing multidimensional weather and climate data into neural networkscited by 31
  3. 2024CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformercited by 22
  4. 2021Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelinescited by 16
  5. 2024FastPersist: Accelerating Model Checkpointing in Deep Learningcited by 6
  6. 2023High Throughput Training of Deep Surrogates from Large Ensemble Runscited by 3
  7. 2023RINAS: Training with Dataset Shuffling Can Be General and Fastcited by 3
  8. 2024I/O in Machine Learning Applications on HPC Systems: A 360-degree Surveycited by 3
  9. 2023KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Trainingcited by 1
  10. 2022SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogatescited by 0
  11. 2023Understanding Patterns of Deep Learning ModelEvolution in Network Architecture Searchcited by 0
  12. 2025RapidGNN: Energy and Communication-Efficient Distributed Training on Large-Scale Graph Neural Networkscited by 0
2 more
  1. 2025Preparation Meets Opportunity: Enhancing Data Preprocessing for ML Training With Senecacited by 0
  2. 2026A principled approach for energy-efficient training via phase-aware GPU frequency tuningcited by 0