# Clairvoyant Prefetching for Distributed Machine Learning I/O

arXiv: https://arxiv.org/abs/2101.08734 · HTML: https://mlsanity.com/paper/2101.08734/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 92 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Data Movement Is All You Need: A Case Study on Optimizing Transformers](https://mlsanity.com/paper/2007.00072/) (2020)
- [Why Random Reshuffling Beats Stochastic Gradient Descent](https://mlsanity.com/paper/1510.08560/) (2015)
- [NVIDIA Tensor Core Programmability, Performance & Precision](https://mlsanity.com/paper/1803.04014/) (2018)
- [Exascale Deep Learning for Climate Analytics](https://mlsanity.com/paper/1810.01993/) (2018)
- [CosmoFlow: Using Deep Learning to Learn the Universe at Scale](https://mlsanity.com/paper/1808.04728/) (2018)
- [The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism](https://mlsanity.com/paper/2007.12856/) (2020)
- [Accelerating Data Loading in Deep Neural Network Training](https://mlsanity.com/paper/1910.01196/) (2019)
- [Characterizing Deep-Learning I/O Workloads in TensorFlow](https://mlsanity.com/paper/1810.03035/) (2018)
- [Detecting Work Zones in SHRP 2 NDS Videos Using Deep Learning Based Computer Vision](https://mlsanity.com/paper/1811.04250/) (2018)
- [Parallelizing Training of Deep Generative Models on Massive Scientific Datasets](https://mlsanity.com/paper/1910.02270/) (2019)
- [Beyond Worst-case Analysis of Multicore Caching Strategies](https://mlsanity.com/paper/2011.02046/) (2020)

## Built on by (most cited first, 14 of 14)

- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Compressing multidimensional weather and climate data into neural networks](https://mlsanity.com/paper/2210.12538/) (2022)
- [CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformer](https://mlsanity.com/paper/2405.03376/) (2024)
- [Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines](https://mlsanity.com/paper/2111.04131/) (2021)
- [FastPersist: Accelerating Model Checkpointing in Deep Learning](https://mlsanity.com/paper/2406.13768/) (2024)
- [High Throughput Training of Deep Surrogates from Large Ensemble Runs](https://mlsanity.com/paper/2309.16743/) (2023)
- [RINAS: Training with Dataset Shuffling Can Be General and Fast](https://mlsanity.com/paper/2312.02368/) (2023)
- [I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey](https://mlsanity.com/paper/2404.10386/) (2024)
- [KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training](https://mlsanity.com/paper/2310.10102/) (2023)
- [SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates](https://mlsanity.com/paper/2211.00224/) (2022)
- [Understanding Patterns of Deep Learning ModelEvolution in Network Architecture Search](https://mlsanity.com/paper/2309.12576/) (2023)
- [RapidGNN: Energy and Communication-Efficient Distributed Training on Large-Scale Graph Neural Networks](https://mlsanity.com/paper/2509.05207/) (2025)
- [Preparation Meets Opportunity: Enhancing Data Preprocessing for ML Training With Seneca](https://mlsanity.com/paper/2511.13724/) (2025)
- [A principled approach for energy-efficient training via phase-aware GPU frequency tuning](https://mlsanity.com/paper/2609.24205/) (2026)

## Abstract (arXiv)

I/O is emerging as a major bottleneck for machine learning training, especially in distributed environments. Indeed, at large scale, I/O takes as much as 85% of training time. Addressing this I/O bottleneck necessitates careful optimization, as optimal data ingestion pipelines differ between systems, and require a delicate balance between access to local storage, external filesystems, and remote nodes. We introduce NoPFS, a machine learning I/O middleware, which provides a scalable, flexible, and easy-to-use solution to the I/O bottleneck. NoPFS uses clairvoyance: Given the seed generating the random access pattern for training with SGD, it can exactly predict when and where a sample will be accessed. We combine this with an analysis of access patterns and a performance model to provide distributed caching policies that adapt to different datasets and storage hierarchies. NoPFS reduces I/O times and improves end-to-end training by up to 5.4x on the ImageNet-1k, ImageNet-22k, and CosmoFlow datasets.