mlsanityData to 7 Oct 2026

Training a Large Video Model on a Single Machine in a Day

arXiv 2309.16669 ↗First posted 28 Sep 2023cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Videos are big, complex to pre-process, and slow to train on. State-of-the-art large-scale video models are trained on clusters of 32 or more GPUs for several days. As a consequence, academia largely ceded the training of large video models to industry. In this paper, we show how to still train a state-of-the-art video model on a single machine with eight consumer-grade GPUs in a day. We identify three bottlenecks, IO, CPU, and GPU computation, and optimize each. The result is a highly efficient video training pipeline. For comparable architectures, our pipeline achieves higher accuracies with frac18 of the computation compared to prior work. Code is available at https://github.com/zhaoyue-zephyrus/AVION.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 80 of its 87 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

80 of 87 references.

References · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
38 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  13. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  16. 2021Video Swin Transformercited by 1,090
  17. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  18. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  19. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  20. 2021Multiscale Vision Transformerscited by 896
  21. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  22. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  24. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  25. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  26. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  27. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  28. 2020Rescaling Egocentric Visioncited by 529
  29. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  30. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  31. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  32. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  33. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  34. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  35. 2022DeiT III: Revenge of the ViTcited by 417
  36. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  37. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  38. 2017Temporal Action Detection with Structured Segment Networkscited by 265

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 8; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  2. 2024Foundation Models for Video Understanding: A Surveycited by 46
  3. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  4. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  5. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  6. 2024Distilling Vision-Language Models on Millions of Videoscited by 20
  7. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  8. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  9. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  10. 2024TIM: A Time Interval Machine for Audio-Visual Action Recognitioncited by 15
  11. 2024Mamba Fusion: Learning Actions Through Questioningcited by 5
  12. 2025EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretrainingcited by 5
8 more
  1. 2024Extending Video Masked Autoencoders to 128 framescited by 3
  2. 2025LLaVAction: evaluating and training multi-modal large language models for action understandingcited by 3
  3. 2025EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimizationcited by 3
  4. 2025D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AIcited by 3
  5. 2024Symmetric Multi-Similarity Loss for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2024cited by 2
  6. 2024FILS: Self-Supervised Video Feature Prediction In Semantic Language Spacecited by 1
  7. 2026MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Datacited by 1
  8. 2025ContextRefine-CLIP for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2025cited by 0