# FFCV: Accelerating Training by Removing Data Bottlenecks

arXiv: https://arxiv.org/abs/2306.12517 · HTML: https://mlsanity.com/paper/2306.12517/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Fixing the train-test resolution discrepancy](https://mlsanity.com/paper/1906.06423/) (2019)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [Leveraging Sparse Linear Layers for Debuggable Deep Networks](https://mlsanity.com/paper/2105.04857/) (2021)
- [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://mlsanity.com/paper/1807.11205/) (2018)
- [tf.data: A Machine Learning Data Processing Framework](https://mlsanity.com/paper/2101.12127/) (2021)
- [Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints](https://mlsanity.com/paper/1905.04753/) (2019)
- [Analyzing and Mitigating Data Stalls in DNN Training](https://mlsanity.com/paper/2007.06775/) (2020)
- [High Performance I/O For Large Scale Deep Learning](https://mlsanity.com/paper/2001.01858/) (2020)
- [Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes](https://mlsanity.com/paper/1711.04325/) (2017)
- [Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark](https://mlsanity.com/paper/1806.01427/) (2018)
- [Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes](https://mlsanity.com/paper/1902.06855/) (2019)
- [ImageNet Training in Minutes](https://mlsanity.com/paper/1709.05011/) (2017)

## Built on by (most cited first, 30 of 49)

- [Graph Metanetworks for Processing Diverse Neural Architectures](https://mlsanity.com/paper/2312.04501/) (2023)
- [Bidirectional Multi-Scale Implicit Neural Representations for Image Deraining](https://mlsanity.com/paper/2404.01547/) (2024)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Scaling MLPs: A Tale of Inductive Bias](https://mlsanity.com/paper/2306.13575/) (2023)
- [Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation](https://mlsanity.com/paper/2503.01776/) (2025)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [94% on CIFAR-10 in 3.29 Seconds on a Single GPU](https://mlsanity.com/paper/2404.00498/) (2024)
- [Optimal Sample Complexity of Contrastive Learning](https://mlsanity.com/paper/2312.00379/) (2023)
- [PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretraining](https://mlsanity.com/paper/2311.12850/) (2023)
- [Understanding the Detrimental Class-level Effects of Data Augmentation](https://mlsanity.com/paper/2401.01764/) (2023)
- [Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods](https://mlsanity.com/paper/2402.04051/) (2024)
- [Disentangling Linear Mode-Connectivity](https://mlsanity.com/paper/2312.09832/) (2023)
- [Structurally Prune Anything: Any Architecture, Any Framework, Any Time](https://mlsanity.com/paper/2403.18955/) (2024)
- [MatMamba: A Matryoshka State Space Model](https://mlsanity.com/paper/2410.06718/) (2024)
- [Do Deep Neural Network Solutions Form a Star Domain?](https://mlsanity.com/paper/2403.07968/) (2024)
- [CSRv2: Unlocking Ultra-Sparse Embeddings](https://mlsanity.com/paper/2602.05735/) (2026)
- [A Quadratic Synchronization Rule for Distributed Deep Learning](https://mlsanity.com/paper/2310.14423/) (2023)
- [A Simple and Efficient Baseline for Data Attribution on Images](https://mlsanity.com/paper/2311.03386/) (2023)
- [Vision Transformers with Natural Language Semantics](https://mlsanity.com/paper/2402.17863/) (2024)
- [Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception](https://mlsanity.com/paper/2405.16493/) (2024)
- [Navigating Scaling Laws: Compute Optimality in Adaptive Model Training](https://mlsanity.com/paper/2311.03233/) (2023)
- [Many Perception Tasks are Highly Redundant Functions of their Input Data](https://mlsanity.com/paper/2407.13841/) (2024)
- [Auditing Approximate Machine Unlearning for Differentially Private Models](https://mlsanity.com/paper/2508.18671/) (2025)
- [D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI](https://mlsanity.com/paper/2510.05684/) (2025)
- [Attributing Learned Concepts in Neural Networks to Training Data](https://mlsanity.com/paper/2310.03149/) (2023)
- [Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configuration](https://mlsanity.com/paper/2310.07555/) (2023)
- [Adaptive Domain Inference Attack with Concept Hierarchy](https://mlsanity.com/paper/2312.15088/) (2023)
- [I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey](https://mlsanity.com/paper/2404.10386/) (2024)
- [LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information](https://mlsanity.com/paper/2412.08771/) (2024)

## Abstract (arXiv)

We present FFCV, a library for easy and fast machine learning model training. FFCV speeds up model training by eliminating (often subtle) data bottlenecks from the training process. In particular, we combine techniques such as an efficient file storage format, caching, data pre-loading, asynchronous data transfer, and just-in-time compilation to (a) make data loading and transfer significantly more efficient, ensuring that GPUs can reach full utilization; and (b) offload as much data processing as possible to the CPU asynchronously, freeing GPU cycles for training. Using FFCV, we train ResNet-18 and ResNet-50 on the ImageNet dataset with competitive tradeoff between accuracy and training time. For example, we are able to train an ImageNet ResNet-50 model to 75% in only 20 mins on a single machine. We demonstrate FFCV's performance, ease-of-use, extensibility, and ability to adapt to resource constraints through several case studies. Detailed installation instructions, documentation, and Slack support channel are available at https://ffcv.io/ .