# Motion-Guided Masking for Spatiotemporal Representation Learning

arXiv: https://arxiv.org/abs/2308.12962 · HTML: https://mlsanity.com/paper/2308.12962/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 30 of 40)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition](https://mlsanity.com/paper/1904.04289/) (2019)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)

## Built on by (most cited first, 30 of 30)

- [Scaling Language-Free Visual Representation Learning](https://mlsanity.com/paper/2504.01017/) (2025)
- [Beyond Language Modeling: An Exploration of Multimodal Pretraining](https://mlsanity.com/paper/2603.03276/) (2026)
- [Causal-JEPA: Learning World Models through Object-Level Latent Masking](https://mlsanity.com/paper/2602.11389/) (2026)
- [Video Token Merging for Long-form Video Understanding](https://mlsanity.com/paper/2410.23782/) (2024)
- [Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders](https://mlsanity.com/paper/2403.17823/) (2024)
- [SIGMA: Sinkhorn-Guided Masked Video Modeling](https://mlsanity.com/paper/2407.15447/) (2024)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [In Pursuit of Pixel Supervision for Visual Pre-training](https://mlsanity.com/paper/2512.15715/) (2025)
- [SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning](https://mlsanity.com/paper/2504.00527/) (2025)
- [Text-Guided Video Masked Autoencoder](https://mlsanity.com/paper/2408.00759/) (2024)
- [Bootstrap Masked Visual Modeling via Hard Patches Mining](https://mlsanity.com/paper/2312.13714/) (2023)
- [SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning](https://mlsanity.com/paper/2504.05706/) (2025)
- [Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation](https://mlsanity.com/paper/2506.11777/) (2025)
- [TESPEC: Temporally-Enhanced Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2508.00913/) (2025)
- [Unfolding Videos Dynamics via Taylor Expansion](https://mlsanity.com/paper/2409.02371/) (2024)
- [Extending Video Masked Autoencoders to 128 frames](https://mlsanity.com/paper/2411.13683/) (2024)
- [LV-MAE: Learning Long Video Representations through Masked-Embedding Autoencoders](https://mlsanity.com/paper/2504.03501/) (2025)
- [TrackMAE: Video Representation Learning via Track Mask and Predict](https://mlsanity.com/paper/2603.27268/) (2026)
- [From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations](https://mlsanity.com/paper/2508.15404/) (2025)
- [Structured-Noise Masked Modeling for Video, Audio and Beyond](https://mlsanity.com/paper/2503.16311/) (2025)
- [Advancing Video Self-Supervised Learning via Image Foundation Models](https://mlsanity.com/paper/2505.19218/) (2025)
- [PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation](https://mlsanity.com/paper/2609.38597/) (2026)
- [Self-Supervised Video Representation Learning in a Heuristic Decoupled Perspective](https://mlsanity.com/paper/2407.14069/) (2024)
- [Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection](https://mlsanity.com/paper/2505.08561/) (2025)
- [ReMA: A Training-Free Plug-and-Play Mixing Augmentation for Video Behavior Recognition](https://mlsanity.com/paper/2601.00311/) (2026)
- [Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis](https://mlsanity.com/paper/2603.25778/) (2026)
- [Entity-Centric World Models: Interaction-Aware Masking for Causal Video Prediction](https://mlsanity.com/paper/2605.15466/) (2026)
- [Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?](https://mlsanity.com/paper/2607.08514/) (2026)
- [JEPA Learns What the Mask Leaves Unrecoverable](https://mlsanity.com/paper/2609.32481/) (2026)
- [Image Classifiers are Efficient Self-Supervised Video Representation Learners](https://mlsanity.com/paper/2609.40347/) (2026)

## Abstract (arXiv)

Several recent works have directly extended the image masked autoencoder (MAE) with random masking into video domain, achieving promising results. However, unlike images, both spatial and temporal information are important for video understanding. This suggests that the random masking strategy that is inherited from the image MAE is less effective for video MAE. This motivates the design of a novel masking algorithm that can more efficiently make use of video saliency. Specifically, we propose a motion-guided masking algorithm (MGM) which leverages motion vectors to guide the position of each mask over time. Crucially, these motion-based correspondences can be directly obtained from information stored in the compressed format of the video, which makes our method efficient and scalable. On two challenging large-scale video benchmarks (Kinetics-400 and Something-Something V2), we equip video MAE with our MGM and achieve up to +1.3% improvement compared to previous state-of-the-art methods. Additionally, our MGM achieves equivalent performance to previous video MAE using up to 66% fewer training epochs. Lastly, we show that MGM generalizes better to downstream transfer learning and domain adaptation tasks on the UCF101, HMDB51, and Diving48 datasets, achieving up to +4.9% improvement compared to baseline methods.