mlsanityData to 7 Oct 2026

Masked Autoencoders As Spatiotemporal Learners

arXiv 2205.09113 ↗First posted 18 May 2022cs.CV cs.LGcited by 428

Not on the map. Why?

Abstract (arXiv)

This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to a large speedup, e.g., > 4x in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training on real-world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT, MAE, etc.) can be a unified methodology for representation learning with minimal domain knowledge.

Key facts. Cited by 428 papers in the map (data through 7 Oct 2026). Builds on 67 of its 86 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

67 of 86 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  14. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  16. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  17. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  18. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  19. 2018Self-Attention with Relative Position Representationscited by 1,112
  20. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  21. 2021Video Swin Transformercited by 1,090
  22. 2016Deep Networks with Stochastic Depthcited by 973
  23. 2021Multiscale Vision Transformerscited by 896
  24. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  25. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  27. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  28. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  29. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  30. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  31. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  32. 2018A Short Note about Kinetics-600cited by 331
  33. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  34. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  35. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  36. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
  37. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  38. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 62; 2023: 137; 2024: 115; 2025: 75; 2026: 39.

428 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  4. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  5. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  6. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  7. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  8. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  9. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  10. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
38 more
  1. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  2. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  3. 2023A Cookbook of Self-Supervised Learningcited by 248
  4. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  5. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  6. 2023Video Understanding with Large Language Models: A Surveycited by 203
  7. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  8. 2023Fast Training of Diffusion Models with Masked Transformerscited by 172
  9. 2022Masked World Models for Visual Controlcited by 169
  10. 2022Recurrent Vision Transformers for Object Detection with Event Camerascited by 158
  11. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  12. 2023Foundation Models for Generalist Geospatial Artificial Intelligencecited by 143
  13. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  14. 2024Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applicationscited by 139
  15. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  16. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 123
  17. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  18. 20234M: Massively Multimodal Masked Modelingcited by 118
  19. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  20. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  21. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  22. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 111
  23. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  24. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  25. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  26. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  27. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  28. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  29. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  30. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  31. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  32. 2023Audio-Visual LLM for Video Understandingcited by 76
  33. 2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 76
  34. 2023Siamese Masked Autoencoderscited by 75
  35. 2023Diffusion Models as Masked Autoencoderscited by 71
  36. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  37. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 67
  38. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66

50 most cited of 428 All 428 (JSON)