mlsanityData to 14 Oct 2026

FFCV: Accelerating Training by Removing Data Bottlenecks

arXiv 2306.12517 ↗First posted 21 Jun 2023cs.LG cs.CVcited by 49

Not on the map. Why?

Abstract (arXiv)

We present FFCV, a library for easy and fast machine learning model training. FFCV speeds up model training by eliminating (often subtle) data bottlenecks from the training process. In particular, we combine techniques such as an efficient file storage format, caching, data pre-loading, asynchronous data transfer, and just-in-time compilation to (a) make data loading and transfer significantly more efficient, ensuring that GPUs can reach full utilization; and (b) offload as much data processing as possible to the CPU asynchronously, freeing GPU cycles for training. Using FFCV, we train ResNet-18 and ResNet-50 on the ImageNet dataset with competitive tradeoff between accuracy and training time. For example, we are able to train an ImageNet ResNet-50 model to 75% in only 20 mins on a single machine. We demonstrate FFCV's performance, ease-of-use, extensibility, and ability to adapt to resource constraints through several case studies. Detailed installation instructions, documentation, and Slack support channel are available at https://ffcv.io/ .

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

25 of 46 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2017Squeeze-and-Excitation Networkscited by 4,789
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  10. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  11. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  12. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 232
13 more
  1. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  2. 2019Fixing the train-test resolution discrepancycited by 209
  3. 2019MLPerf Training Benchmarkcited by 115
  4. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  5. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  6. 2021tf.data: A Machine Learning Data Processing Frameworkcited by 41
  7. 2019Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraintscited by 37
  8. 2020Analyzing and Mitigating Data Stalls in DNN Trainingcited by 36
  9. 2020High Performance I/O For Large Scale Deep Learningcited by 26
  10. 2017Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutescited by 22
  11. 2018Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmarkcited by 21
  12. 2019Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutescited by 13
  13. 2017ImageNet Training in Minutescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 19; 2025: 11; 2026: 5.

49 papers.

Sort:

By area:

  1. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 57
  2. 2024Bidirectional Multi-Scale Implicit Neural Representations for Image Derainingcited by 54
  3. 2024Towards Scalable and Versatile Weight Space Learningcited by 41
  4. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  5. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 26
  6. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  7. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  8. 202494% on CIFAR-10 in 3.29 Seconds on a Single GPUcited by 16
  9. 2023Optimal Sample Complexity of Contrastive Learningcited by 15
  10. 2023PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretrainingcited by 13
  11. 2023Understanding the Detrimental Class-level Effects of Data Augmentationcited by 13
  12. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 9
37 more
  1. 2023Disentangling Linear Mode-Connectivitycited by 8
  2. 2024Structurally Prune Anything: Any Architecture, Any Framework, Any Timecited by 8
  3. 2024MatMamba: A Matryoshka State Space Modelcited by 7
  4. 2024Do Deep Neural Network Solutions Form a Star Domain?cited by 6
  5. 2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 6
  6. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  7. 2023A Simple and Efficient Baseline for Data Attribution on Imagescited by 5
  8. 2024Vision Transformers with Natural Language Semanticscited by 5
  9. 2024Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perceptioncited by 5
  10. 2023Navigating Scaling Laws: Compute Optimality in Adaptive Model Trainingcited by 4
  11. 2024Many Perception Tasks are Highly Redundant Functions of their Input Datacited by 4
  12. 2025Auditing Approximate Machine Unlearning for Differentially Private Modelscited by 4
  13. 2025D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AIcited by 4
  14. 2023Attributing Learned Concepts in Neural Networks to Training Datacited by 3
  15. 2023Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configurationcited by 3
  16. 2023Adaptive Domain Inference Attack with Concept Hierarchycited by 3
  17. 2024I/O in Machine Learning Applications on HPC Systems: A 360-degree Surveycited by 3
  18. 2024LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Informationcited by 3
  19. 2025First-Passage Approach to Optimizing Perturbations for Improved Training of Machine Learning Modelscited by 3
  20. 2025Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machinescited by 3
  21. 2023Efficient Rotation Invariance in Deep Neural Networks through Artificial Mental Rotationcited by 2
  22. 2024DailyMAE: Towards Pretraining Masked Autoencoders in One Daycited by 2
  23. 2024MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representationscited by 2
  24. 2025Coreset Selection via LLM-based Concept Bottleneckscited by 2
  25. 2026FOVI: A biologically-inspired foveated interface for deep vision modelscited by 2
  26. 2024A Perspective on Deep Vision Performance with Standard Image and Video Codecscited by 1
  27. 2024Cyclic Sparse Training: Is it Enough?cited by 1
  28. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1
  29. 2024FT-PrivacyScore: Personalized Privacy Scoring Service for Machine Learning Participationcited by 1
  30. 2025MegaFold: Efficient Training of Next-Generation 3D Attention Protein Models on Cross-Platform GPUscited by 1
  31. 2025A Mutual Information Perspective on Multiple Latent Variable Generative Models for Positive View Generationcited by 0
  32. 2025Evolutionary Prediction Gamescited by 0
  33. 2025Neural Architecture Search by Learning a Hierarchical Search Spacecited by 0
  34. 2025Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Datacited by 0
  35. 2026GetBatch: Distributed Multi-Object Retrieval for ML Data Loadingcited by 0
  36. 2026Choosing a JPEG Decoder for PyTorch DataLoaders: Workload-Specific Throughput on Four CPUscited by 0
  37. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0