mlsanityData to 7 Oct 2026

VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

arXiv 2303.16727 ↗First posted 29 Mar 2023cs.CV cs.LGcited by 516

Not on the map. Why?

Cited by 29 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner. The code and model is available at urlhttps://github.com/OpenGVLab/VideoMAEv2.

Key facts. Cited by 516 papers in the map (data through 7 Oct 2026). Builds on 103 of its 122 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

103 of 122 references.

References · 103

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2017Non-local Neural Networkscited by 2,290
  13. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  14. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  15. 2017The Kinetics Human Action Video Datasetcited by 1,894
  16. 2018SlowFast Networks for Video Recognitioncited by 1,862
  17. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  18. 2021ViViT: A Video Vision Transformercited by 1,610
  19. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  20. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  21. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  22. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  23. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  24. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  25. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  26. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  27. 2021Video Swin Transformercited by 1,090
  28. 2021Scaling Vision Transformerscited by 993
  29. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  30. 2021Multiscale Vision Transformerscited by 896
  31. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  32. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  33. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  34. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  35. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  36. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  37. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  38. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 62; 2024: 146; 2025: 139; 2026: 169.

516 papers.

Sort:

By area:

  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  3. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  4. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  5. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  7. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  8. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  9. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  10. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  11. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  12. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
38 more
  1. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  2. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  3. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  4. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  5. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  6. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  7. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  8. 2024Streaming Dense Video Captioningcited by 82
  9. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  10. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  11. 2024Progressive Autoregressive Video Diffusion Modelscited by 65
  12. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  13. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
  14. 2024On the Content Bias in Fr'echet Video Distancecited by 51
  15. 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 49
  16. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 48
  17. 2023End-to-End Temporal Action Detection with 1B Parameters Across 1000 Framescited by 47
  18. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  19. 2025Efficiently Reconstructing Dynamic Scenes One D4RT at a Timecited by 46
  20. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  21. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  22. 2024DINO-Foresight: Looking into the Future with DINOcited by 44
  23. 2024Towards Privacy-Aware Sign Language Translation at Scalecited by 42
  24. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  25. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 41
  26. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  27. 2023MGMAE: Motion Guided Masking for Video Masked Autoencodingcited by 40
  28. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  29. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  30. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  31. 2024GAIA: Rethinking Action Quality Assessment for AI-Generated Videoscited by 30
  32. 2024Scaling 4D Representationscited by 30
  33. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  34. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  35. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  36. 2024Masked Image Modeling: A Surveycited by 28
  37. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  38. 2025MaskGWM: A Generalizable Driving World Model with Video Mask Reconstructioncited by 27

50 most cited of 516 All 516 (JSON)