mlsanityData to 13 Oct 2026

MaskViT: Masked Visual Pre-Training for Video Prediction

arXiv 2206.11894 ↗First posted 23 Jun 2022cs.CV cs.LG cs.ROcited by 113

Not on the map. Why?

Cited by 23 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to predict future visual observations conditioned on past observations and motor commands can enable embodied agents to plan solutions to a variety of tasks in complex environments. This work shows that we can create good video prediction models by pre-training transformers via masked visual modeling. Our approach, named MaskViT, is based on two simple design decisions. First, for memory and training efficiency, we use two types of window attention: spatial and spatiotemporal. Second, during training, we mask a variable percentage of tokens instead of a fixed mask ratio. For inference, MaskViT generates all tokens via iterative refinement where we incrementally decrease the masking ratio following a mask scheduling function. On several datasets we demonstrate that MaskViT outperforms prior works in video prediction, is parameter efficient, and can generate high-resolution videos (256x256). Further, we demonstrate the benefits of inference speedup (up to 512x) due to iterative decoding by using MaskViT for planning on a real robot. Our work suggests that we can endow embodied agents with powerful predictive models by leveraging the general framework of masked visual modeling with minimal domain knowledge.

Key facts. Cited by 113 papers in the map (data through 13 Oct 2026). Builds on 67 of its 81 references that are in the map. First posted on arXiv 23 Jun 2022.

Builds on

67 of 81 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  9. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  12. 2017Neural Discrete Representation Learningcited by 4,680
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,751
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,687
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  6. 2022Video Diffusion Modelscited by 2,293
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,535
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,488
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,381
  12. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,308
  13. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,217
  14. 2021Video Swin Transformercited by 1,090
  15. 2022MaskGIT: Masked Generative Image Transformercited by 1,090
  16. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 747
  17. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  18. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 683
  19. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 653
  20. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  21. 2017Non-Autoregressive Neural Machine Translationcited by 462
  22. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  23. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  24. 2016Deep Visual Foresight for Planning Robot Motioncited by 281
  25. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  26. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 259
  27. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  28. 2022Masked Visual Pre-training for Motor Controlcited by 246
  29. 2018Stochastic Video Generation with a Learned Priorcited by 245
  30. 2017Stochastic Variational Video Predictioncited by 234
  31. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 224
  32. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  33. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 201
  34. 2019Scaling Autoregressive Video Modelscited by 190
  35. 2018Stochastic Adversarial Video Predictioncited by 173
  36. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 163
  37. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  38. 2017Video Generation From Textcited by 154

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 13; 2023: 43; 2024: 28; 2025: 19; 2026: 10.

113 papers.

Sort:

By area:

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,768
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 769
  3. 2024Genie: Generative Interactive Environmentscited by 721
  4. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 636
  5. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 482
  6. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 473
  7. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 357
  8. 2022MAGVIT: Masked Generative Video Transformercited by 343
  9. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 341
  10. 2023Photorealistic Video Generation with Diffusion Modelscited by 305
  11. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 293
  12. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
38 more
  1. 2023Video Understanding with Large Language Models: A Surveycited by 205
  2. 2022Masked World Models for Visual Controlcited by 170
  3. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  4. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 140
  5. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 109
  6. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  7. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  8. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 95
  9. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 76
  10. 2023Siamese Masked Autoencoderscited by 75
  11. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  12. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 55
  13. 2022Heterogeneous Graph Masked Autoencoderscited by 47
  14. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  15. 2024DINO-Foresight: Looking into the Future with DINOcited by 45
  16. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  17. 2022Temporally Consistent Transformers for Video Generationcited by 43
  18. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  19. 2024Mora: Enabling Generalist Video Generation via A Multi-Agent Frameworkcited by 42
  20. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 41
  21. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  22. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  23. 2024Learning Manipulation by Predicting Interactioncited by 36
  24. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  25. 2023Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesiscited by 32
  26. 2023TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Drivingcited by 30
  27. 2023Coarse-to-Fine Amodal Segmentation with Shape Priorcited by 29
  28. 2024Masked Image Modeling: A Surveycited by 28
  29. 2023Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modelingcited by 25
  30. 2023A Control-Centric Benchmark for Video Predictioncited by 25
  31. 2023Hierarchical Masked 3D Diffusion Model for Video Outpaintingcited by 25
  32. 2024General surgery vision transformer: A video pre-trained foundation model for general surgerycited by 25
  33. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  34. 2024VideoMAC: Video Masked Autoencoders Meet ConvNetscited by 20
  35. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  36. 2023Is Imitation All You Need? Generalized Decision-Making with Dual-Phase Trainingcited by 18
  37. 2023MMVP: Motion-Matrix-based Video Predictioncited by 18
  38. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MaskWAM: Unifying Mask Prompting and Prediction for World-Action Models
  2. 2026Masked Visual Actions for Unified World Modeling
  3. 2026SelfWAM: A Self-Grounded Unified World Action Model for Fast Robot Control
  4. 2026Mask World Model: Predicting What Matters for Robust Robot Policy Learning
  5. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
  6. 2026MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Prediction
  7. 2025InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision
  8. 2024Predicting Long-horizon Futures by Conditioning on Geometry and Time