# MGMAE: Motion Guided Masking for Video Masked Autoencoding

arXiv: https://arxiv.org/abs/2308.10794 · HTML: https://mlsanity.com/paper/2308.10794/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 21 Aug 2023.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)

## Built on by (most cited first, 30 of 41)

- [vid-TLDR: Training Free Token merging for Light-weight Video Transformer](https://mlsanity.com/paper/2403.13347/) (2024)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [SIGMA: Sinkhorn-Guided Masked Video Modeling](https://mlsanity.com/paper/2407.15447/) (2024)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [SMC++: Masked Learning of Unsupervised Video Semantic Compression](https://mlsanity.com/paper/2406.04765/) (2024)
- [T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learning](https://mlsanity.com/paper/2312.10217/) (2023)
- [SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning](https://mlsanity.com/paper/2504.00527/) (2025)
- [FocusMAE: Gallbladder Cancer Detection from Ultrasound Videos with Focused Masked Autoencoders](https://mlsanity.com/paper/2403.08848/) (2024)
- [Text-Guided Video Masked Autoencoder](https://mlsanity.com/paper/2408.00759/) (2024)
- [Universal Visuo-Tactile Video Understanding for Embodied Interaction](https://mlsanity.com/paper/2505.22566/) (2025)
- [MV2MAE: Multi-View Video Masked Autoencoders](https://mlsanity.com/paper/2401.15900/) (2024)
- [Bootstrap Masked Visual Modeling via Hard Patches Mining](https://mlsanity.com/paper/2312.13714/) (2023)
- [SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning](https://mlsanity.com/paper/2504.05706/) (2025)
- [Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation](https://mlsanity.com/paper/2506.11777/) (2025)
- [TESPEC: Temporally-Enhanced Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2508.00913/) (2025)
- [Extending Video Masked Autoencoders to 128 frames](https://mlsanity.com/paper/2411.13683/) (2024)
- [LV-MAE: Learning Long Video Representations through Masked-Embedding Autoencoders](https://mlsanity.com/paper/2504.03501/) (2025)
- [Simplifying Traffic Anomaly Detection with Video Foundation Models](https://mlsanity.com/paper/2507.09338/) (2025)
- [Self-Guided Masked Autoencoder](https://mlsanity.com/paper/2507.19773/) (2025)
- [TrackMAE: Video Representation Learning via Track Mask and Predict](https://mlsanity.com/paper/2603.27268/) (2026)
- [Asymmetric Masked Distillation for Pre-Training Small Foundation Models](https://mlsanity.com/paper/2311.03149/) (2023)
- [Dynamic Traceback Learning for Medical Report Generation](https://mlsanity.com/paper/2401.13267/) (2024)
- [Predicting Long-horizon Futures by Conditioning on Geometry and Time](https://mlsanity.com/paper/2404.11554/) (2024)
- [CrowdMAC: Masked Crowd Density Completion for Robust Crowd Density Forecasting](https://mlsanity.com/paper/2407.14725/) (2024)
- [Structured-Noise Masked Modeling for Video, Audio and Beyond](https://mlsanity.com/paper/2503.16311/) (2025)
- [Fine-Tuning Video Transformers for Word-Level Bangla Sign Language: A Comparative Analysis for Classification Tasks](https://mlsanity.com/paper/2506.04367/) (2025)
- [Dynamic and Compressive Adaptation of Transformers From Images to Videos](https://mlsanity.com/paper/2408.06840/) (2024)
- [KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder](https://mlsanity.com/paper/2411.12270/) (2024)
- [Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection](https://mlsanity.com/paper/2505.08561/) (2025)
- [Semantically Informed Salient Regions Guided Radiology Report Generation](https://mlsanity.com/paper/2507.11015/) (2025)

## Abstract (arXiv)

Masked autoencoding has shown excellent performance on self-supervised video representation learning. Temporal redundancy has led to a high masking ratio and customized masking strategy in VideoMAE. In this paper, we aim to further improve the performance of video masked autoencoding by introducing a motion guided masking strategy. Our key insight is that motion is a general and unique prior in video, which should be taken into account during masked pre-training. Our motion guided masking explicitly incorporates motion information to build temporal consistent masking volume. Based on this masking volume, we can track the unmasked tokens in time and sample a set of temporal consistent cubes from videos. These temporal aligned unmasked tokens will further relieve the information leakage issue in time and encourage the MGMAE to learn more useful structure information. We implement our MGMAE with an online efficient optical flow estimator and backward masking map warping strategy. We perform experiments on the datasets of Something-Something V2 and Kinetics-400, demonstrating the superior performance of our MGMAE to the original VideoMAE. In addition, we provide the visualization analysis to illustrate that our MGMAE can sample temporal consistent cubes in a motion-adaptive manner for more effective video pre-training.