mlsanityData to 13 Oct 2026

VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

arXiv 2203.12602 ↗First posted 23 Mar 2022cs.CVcited by 1,308

Not on the map. Why?

Cited by 56 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking with an extremely high ratio. This simple design makes video reconstruction a more challenging self-supervision task, thus encouraging extracting more effective video representations during this pre-training process. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables a higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets is an important issue. Notably, our VideoMAE with the vanilla ViT can achieve 87.4% on Kinetics-400, 75.4% on Something-Something V2, 91.3% on UCF101, and 62.6% on HMDB51, without using any extra data. Code is available at https://github.com/MCG-NJU/VideoMAE.

Key facts. Cited by 1,308 papers in the map (data through 13 Oct 2026). Builds on 78 of its 108 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

78 of 108 references.

References · 78

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  8. 2020End-to-End Object Detection with Transformerscited by 7,396
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,457
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,264
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,930
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  8. 2017Non-local Neural Networkscited by 2,291
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,266
  10. 2017The Kinetics Human Action Video Datasetcited by 1,900
  11. 2018SlowFast Networks for Video Recognitioncited by 1,864
  12. 2021ViViT: A Video Vision Transformercited by 1,615
  13. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,614
  14. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,427
  16. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,150
  17. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  18. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,093
  19. 2021Video Swin Transformercited by 1,090
  20. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  21. 2021Multiscale Vision Transformerscited by 896
  22. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 895
  23. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 853
  24. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 653
  25. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 574
  27. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  28. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  29. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  30. 2021Transformer Trackingcited by 412
  31. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  32. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  33. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  34. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  35. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 279
  36. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  37. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 265
  38. 2020Self-supervised Co-training for Video Representation Learningcited by 232

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 90; 2023: 242; 2024: 316; 2025: 346; 2026: 314.

1,308 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,168
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 854
  4. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 715
  5. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 689
  6. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 540
  7. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 520
  8. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 514
  9. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 492
  10. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 482
  11. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 462
  12. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 446
38 more
  1. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  2. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 406
  3. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  4. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  5. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 311
  6. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 288
  7. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 251
  8. 2023A Cookbook of Self-Supervised Learningcited by 249
  9. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 242
  10. 2023FuXi: A cascade machine learning forecasting system for 15-day global weather forecastcited by 240
  11. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 227
  12. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  13. 2023Video Understanding with Large Language Models: A Surveycited by 205
  14. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  15. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  16. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  17. 2022Masked World Models for Visual Controlcited by 170
  18. 2024Streaming Long Video Understanding with Large Language Modelscited by 166
  19. 2023Language-Driven Representation Learning for Roboticscited by 163
  20. 2022Recurrent Vision Transformers for Object Detection with Event Camerascited by 159
  21. 2023Foundation Models for Generalist Geospatial Artificial Intelligencecited by 147
  22. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 146
  23. 2022Contrastive Audio-Visual Masked Autoencodercited by 144
  24. 2024Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applicationscited by 143
  25. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  26. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  27. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  28. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  29. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  30. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  31. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 118
  32. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  33. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  34. 2023FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimationcited by 114
  35. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 113
  36. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 113
  37. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  38. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 108

50 most cited of 1,308 All 1,308 (JSON)