mlsanityData to 14 Oct 2026

MAGVIT: Masked Generative Video Transformer

arXiv 2212.05199 ↗First posted 10 Dec 2022cs.CVcited by 346

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spatial-temporal visual tokens and propose an embedding method for masked video token modeling to facilitate multi-task learning. We conduct extensive experiments to demonstrate the quality, efficiency, and flexibility of MAGVIT. Our experiments show that (i) MAGVIT performs favorably against state-of-the-art approaches and establishes the best-published FVD on three video generation benchmarks, including the challenging Kinetics-600. (ii) MAGVIT outperforms existing methods in inference time by two orders of magnitude against diffusion models and by 60x against autoregressive models. (iii) A single MAGVIT model supports ten diverse generation tasks and generalizes across videos from different visual domains. The source code and trained models will be released to the public at https://magvit.cs.cmu.edu.

Key facts. Cited by 346 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 70 of its 86 references that are in the map. First posted on arXiv 10 Dec 2022.

Builds on

70 of 86 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022MaskGIT: Masked Generative Image Transformer8 Feb
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  3. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale22 Oct
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  6. 2022Video Diffusion Modelscited by 2,305
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  9. 2018Group Normalizationcited by 1,524
  10. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  13. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  14. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  15. 2016Pixel Recurrent Neural Networkscited by 816
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  17. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  18. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  19. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  20. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  21. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  22. 2016Generating Videos with Scene Dynamicscited by 376
  23. 2018A Short Note about Kinetics-600cited by 333
  24. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  25. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  26. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  27. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  28. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  29. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  30. 2019Scaling Autoregressive Video Modelscited by 190
  31. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  32. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 176
  33. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  34. 2020Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotationscited by 155
  35. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  36. 2022Diffusion Models for Video Prediction and Infillingcited by 135
  37. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  38. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 41; 2024: 118; 2025: 126; 2026: 61.

346 papers.

Sort:

By area:

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  4. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  5. 2024Genie: Generative Interactive Environmentscited by 747
  6. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  7. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  8. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  9. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  10. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  11. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  12. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
38 more
  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  2. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  3. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  4. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
  5. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  6. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  7. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  8. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  9. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  10. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  11. 2024Navigation World Modelscited by 272
  12. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  13. 2023DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimizationcited by 221
  14. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  15. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 198
  16. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
  17. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  18. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 185
  19. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 183
  20. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  21. 2025History-Guided Video Diffusioncited by 171
  22. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167
  23. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 166
  24. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  25. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 164
  26. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  27. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
  28. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  29. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  30. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  31. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 146
  32. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 132
  33. 2024Randomized Autoregressive Visual Generationcited by 127
  34. 2025Long Context Tuning for Video Generationcited by 113
  35. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 112
  36. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  37. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 106
  38. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105

50 most cited of 346 All 346 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Multi-Task Multi-Modal Models: A Video Generative Perspective
  2. 2024Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer
  3. 2025Mask2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation
  4. 2025REGEN: Learning Compact Video Embedding with (Re-)Generative Decoder
  5. 2025Training-Free Efficient Video Generation via Dynamic Token Carving
  6. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
  7. 2024Motion Guided Token Compression for Efficient Masked Video Modeling
  8. 2025Learnings from Scaling Visual Tokenizers for Reconstruction and Generation