mlsanityData to 7 Oct 2026

Turbo Training with Token Dropout

arXiv 2210.04889 ↗First posted 10 Oct 2022cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

The objective of this paper is an efficient training method for video tasks. We make three contributions: (1) We propose Turbo training, a simple and versatile training paradigm for Transformers on multiple video tasks. (2) We illustrate the advantages of Turbo training on action classification, video-language representation learning, and long-video activity classification, showing that Turbo training can largely maintain competitive performance while achieving almost 4X speed-up and significantly less memory consumption. (3) Turbo training enables long-schedule video-language training and end-to-end long-video training, delivering competitive or superior performance than previous works, which were infeasible to train under limited resources.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 53 of its 61 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

53 of 61 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  2. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  13. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  14. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  17. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  18. 2016Deep Networks with Stochastic Depthcited by 973
  19. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  20. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  21. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  22. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  23. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  24. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  25. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  26. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  27. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  28. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  29. 2020Sparse Sinkhorn Attentioncited by 250
  30. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  31. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  32. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133
  33. 2019Large Memory Layers with Product Keyscited by 120
  34. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  35. 2017Compressed Video Action Recognitioncited by 102
  36. 2018Timeception for Complex Action Recognitioncited by 89
  37. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  38. 2022Temporal Alignment Networks for Long-term Videocited by 84

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 7; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  2. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  3. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  4. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  5. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  6. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 10
  7. 2025ELIP: Enhanced Visual-Language Foundation Models for Image Retrievalcited by 9
  8. 2025Make Your Training Flexible: Towards Deployment-Efficient Video Modelscited by 6
  9. 2025MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognitioncited by 4
  10. 2025VideoMAP: Toward Scalable Mamba-based Video Autoregressive Pretrainingcited by 3
  11. 2025LV-MAE: Learning Long Video Representations through Masked-Embedding Autoencoderscited by 3
  12. 2025Video, How Do Your Tokens Merge?cited by 2
2 more
  1. 2023Harvest Video Foundation Models via Efficient Post-Pretrainingcited by 1
  2. 2025CRAM: Large-scale Video Continual Learning with Bootstrapped Compressioncited by 1