# Large-scale weakly-supervised pre-training for video action recognition

arXiv: https://arxiv.org/abs/1905.00561 · HTML: https://mlsanity.com/paper/1905.00561/

## Key facts

- Cited by 130 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2019.

## Built on by (most cited first, 30 of 130)

- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Self-Supervised Pretraining of 3D Features on any Point-Cloud](https://mlsanity.com/paper/2101.02691/) (2021)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [Multiview Transformers for Video Recognition](https://mlsanity.com/paper/2201.04288/) (2022)
- [StyleGAN-Human: A Data-Centric Odyssey of Human Generation](https://mlsanity.com/paper/2204.11823/) (2022)
- [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [Learning To Recognize Procedural Activities with Distant Supervision](https://mlsanity.com/paper/2201.10990/) (2022)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [Bridging Video-text Retrieval with Multiple Choice Questions](https://mlsanity.com/paper/2201.04850/) (2022)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [Elaborative Rehearsal for Zero-shot Action Recognition](https://mlsanity.com/paper/2108.02833/) (2021)
- [Verbs in Action: Improving verb understanding in video-language models](https://mlsanity.com/paper/2304.06708/) (2023)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [Revisiting Classifier: Transferring Vision-Language Models for Video Recognition](https://mlsanity.com/paper/2207.01297/) (2022)
- [The effectiveness of MAE pre-pretraining for billion-scale pretraining](https://mlsanity.com/paper/2303.13496/) (2023)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning](https://mlsanity.com/paper/2106.11250/) (2021)
- [InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges](https://mlsanity.com/paper/2211.09529/) (2022)
- [Procedure-Aware Pretraining for Instructional Video Understanding](https://mlsanity.com/paper/2303.18230/) (2023)
- [Forecasting Action through Contact Representations from First Person Video](https://mlsanity.com/paper/2102.00649/) (2021)
- [Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations](https://mlsanity.com/paper/2303.17839/) (2023)
- [VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs](https://mlsanity.com/paper/2101.12059/) (2021)
- [Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models](https://mlsanity.com/paper/2103.08849/) (2021)
- [AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition](https://mlsanity.com/paper/2112.14238/) (2021)
- [Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition](https://mlsanity.com/paper/2110.10101/) (2021)
- [HowToCaption: Prompting LLMs to Transform Video Annotations at Scale](https://mlsanity.com/paper/2310.04900/) (2023)

## Abstract (arXiv)

Current fully-supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain-specific labels. This hinders the progress towards advanced video architectures. This paper presents an in-depth study of using large volumes of web videos for pre-training video models for the task of action recognition. Our primary empirical finding is that pre-training at a very large scale (over 65 million videos), despite on noisy social-media videos and hashtags, substantially improves the state-of-the-art on three challenging public action recognition datasets. Further, we examine three questions in the construction of weakly-supervised video action datasets. First, given that actions involve interactions with objects, how should one construct a verb-object pre-training label space to benefit transfer learning the most? Second, frame-based models perform quite well on action recognition; is pre-training for good image features sufficient or is pre-training for spatio-temporal features valuable for optimal transfer learning? Finally, actions are generally less well-localized in long videos vs. short videos; since action labels are provided at a video level, how should one choose video clips for best performance, given some fixed budget of number or minutes of videos?