mlsanityData to 14 Oct 2026

MGMAE: Motion Guided Masking for Video Masked Autoencoding

arXiv 2308.10794 ↗First posted 21 Aug 2023cs.CV cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

Masked autoencoding has shown excellent performance on self-supervised video representation learning. Temporal redundancy has led to a high masking ratio and customized masking strategy in VideoMAE. In this paper, we aim to further improve the performance of video masked autoencoding by introducing a motion guided masking strategy. Our key insight is that motion is a general and unique prior in video, which should be taken into account during masked pre-training. Our motion guided masking explicitly incorporates motion information to build temporal consistent masking volume. Based on this masking volume, we can track the unmasked tokens in time and sample a set of temporal consistent cubes from videos. These temporal aligned unmasked tokens will further relieve the information leakage issue in time and encourage the MGMAE to learn more useful structure information. We implement our MGMAE with an online efficient optical flow estimator and backward masking map warping strategy. We perform experiments on the datasets of Something-Something V2 and Kinetics-400, demonstrating the superior performance of our MGMAE to the original VideoMAE. In addition, we provide the visualization analysis to illustrate that our MGMAE can sample temporal consistent cubes in a motion-adaptive manner for more effective video pre-training.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 49 of its 63 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

49 of 63 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2017Non-local Neural Networkscited by 2,293
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  12. 2017The Kinetics Human Action Video Datasetcited by 1,909
37 more
  1. 2018SlowFast Networks for Video Recognitioncited by 1,867
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  9. 2021Video Swin Transformercited by 1,091
  10. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  11. 2021Multiscale Vision Transformerscited by 897
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  13. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  14. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  15. 2021BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignmentcited by 342
  16. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  17. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  18. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  19. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 281
  20. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  21. 2019Depth-Aware Video Frame Interpolationcited by 234
  22. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  23. 2020TDN: Temporal Difference Networks for Efficient Action Recognitioncited by 183
  24. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  25. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  26. 2020TAM: Temporal Adaptive Module for Video Recognitioncited by 139
  27. 2021Relaxed Transformer Decoders for Direct Action Proposal Generationcited by 117
  28. 2023Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolationcited by 98
  29. 2019TEINet: Towards an Efficient Architecture for Video Recognitioncited by 92
  30. 2020MotionSqueeze: Neural Motion Feature Learning for Video Understandingcited by 72
  31. 2022Flow-Guided Transformer for Video Inpaintingcited by 59
  32. 2015Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptorscited by 58
  33. 2021VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learningcited by 56
  34. 2021MGSampler: An Explainable Sampling Strategy for Video Action Recognitioncited by 49
  35. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 43
  36. 2022MAR: Masked Autoencoders for Efficient Action Recognitioncited by 29
  37. 2020Hierarchical Contrastive Motion Learning for Video Action Recognitioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 14; 2026: 10.

41 papers.

Sort:

By area:

  1. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  2. 2024Masked Image Modeling: A Surveycited by 29
  3. 2024SIGMA: Sinkhorn-Guided Masked Video Modelingcited by 14
  4. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  5. 2024SMC++: Masked Learning of Unsupervised Video Semantic Compressioncited by 12
  6. 2023T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learningcited by 11
  7. 2025SMILE: Infusing Spatial and Motion Semantics in Masked Video Learningcited by 9
  8. 2024FocusMAE: Gallbladder Cancer Detection from Ultrasound Videos with Focused Masked Autoencoderscited by 8
  9. 2024Text-Guided Video Masked Autoencodercited by 8
  10. 2025Universal Visuo-Tactile Video Understanding for Embodied Interactioncited by 8
  11. 2024MV2MAE: Multi-View Video Masked Autoencoderscited by 6
  12. 2023Bootstrap Masked Visual Modeling via Hard Patches Miningcited by 5
29 more
  1. 2025SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learningcited by 4
  2. 2025Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillationcited by 4
  3. 2025TESPEC: Temporally-Enhanced Self-Supervised Pretraining for Event Camerascited by 4
  4. 2024Extending Video Masked Autoencoders to 128 framescited by 3
  5. 2025LV-MAE: Learning Long Video Representations through Masked-Embedding Autoencoderscited by 3
  6. 2025Simplifying Traffic Anomaly Detection with Video Foundation Modelscited by 3
  7. 2025Self-Guided Masked Autoencodercited by 3
  8. 2026TrackMAE: Video Representation Learning via Track Mask and Predictcited by 3
  9. 2023Asymmetric Masked Distillation for Pre-Training Small Foundation Modelscited by 2
  10. 2024Dynamic Traceback Learning for Medical Report Generationcited by 1
  11. 2024Predicting Long-horizon Futures by Conditioning on Geometry and Timecited by 1
  12. 2024CrowdMAC: Masked Crowd Density Completion for Robust Crowd Density Forecastingcited by 1
  13. 2025Structured-Noise Masked Modeling for Video, Audio and Beyondcited by 1
  14. 2025Fine-Tuning Video Transformers for Word-Level Bangla Sign Language: A Comparative Analysis for Classification Taskscited by 1
  15. 2024Dynamic and Compressive Adaptation of Transformers From Images to Videoscited by 0
  16. 2024KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encodercited by 0
  17. 2025Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selectioncited by 0
  18. 2025Semantically Informed Salient Regions Guided Radiology Report Generationcited by 0
  19. 2025A Renaissance of Explicit Motion Information Mining from Transformers for Action Recognitioncited by 0
  20. 2025A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Rankingcited by 0
  21. 2026Respiratory Status Detection with Video Transformerscited by 0
  22. 2026Entity-Centric World Models: Interaction-Aware Masking for Causal Video Predictioncited by 0
  23. 2026Reweighting Framewise Attention in Video Transformers for Facial Expression Understandingcited by 0
  24. 2026Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?cited by 0
  25. 2026Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Drivingcited by 0
  26. 2026Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Camerascited by 0
  27. 2026JEPA Learns What the Mask Leaves Unrecoverablecited by 0
  28. 2026Image Classifiers are Efficient Self-Supervised Video Representation Learnerscited by 0
  29. 2026Video-Based Sexual Harassment Detection Using Fine-Tuned Video Transformerscited by 0