mlsanityData to 14 Oct 2026

VideoGPT: Video Generation using VQ-VAE and Transformers

arXiv 2104.10157 ↗First posted 20 Apr 2021cs.CV cs.LGcited by 658

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html

Key facts. Cited by 658 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 65 of its 77 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

65 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2016Layer Normalizationcited by 5,669
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2013Auto-Encoding Variational Bayescited by 4,053
  11. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  12. 2016Improved Techniques for Training GANscited by 2,970
38 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  2. 2017Non-local Neural Networkscited by 2,293
  3. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  4. 2016Density estimation using Real NVPcited by 2,042
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  7. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  10. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  11. 2016Pixel Recurrent Neural Networkscited by 816
  12. 2018Self-Attention Generative Adversarial Networkscited by 809
  13. 2018Image Transformercited by 772
  14. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
  15. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  16. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  17. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  18. 2020Jukebox: A Generative Model for Musiccited by 542
  19. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  20. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  21. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  22. 2018Video-to-Video Synthesiscited by 390
  23. 2016Generating Videos with Scene Dynamicscited by 376
  24. 2019Axial Attention in Multidimensional Transformerscited by 358
  25. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  26. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 264
  27. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 262
  28. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  29. 2018Stochastic Video Generation with a Learned Priorcited by 245
  30. 2017Stochastic Variational Video Predictioncited by 234
  31. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  32. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  33. 2019Scaling Autoregressive Video Modelscited by 190
  34. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  35. 2018Stochastic Adversarial Video Predictioncited by 173
  36. 2017Unsupervised Learning of Disentangled Representations from Videocited by 139
  37. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  38. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 66; 2023: 105; 2024: 177; 2025: 166; 2026: 127.

658 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2022Video Diffusion Modelscited by 2,305
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  7. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  8. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  9. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  10. 2024Genie: Generative Interactive Environmentscited by 747
  11. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  12. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
38 more
  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  2. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  3. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  4. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  5. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  6. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  7. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  8. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  9. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  10. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  11. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  12. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  13. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  14. 2022MAGVIT: Masked Generative Video Transformercited by 346
  15. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  16. 2022Transformers are Sample-Efficient World Modelscited by 328
  17. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  18. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  19. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  20. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
  21. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  22. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  23. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  24. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  25. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  26. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  27. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 250
  28. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  29. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  30. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
  31. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 233
  32. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 212
  33. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  34. 2023A Survey on Video Diffusion Modelscited by 206
  35. 2025Unified Video Action Modelcited by 205
  36. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 198
  37. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  38. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187

50 most cited of 658 All 658 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Image Generation Models: A Technical History
  2. 2026Video Generation with Predictive Latents
  3. 2024ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Models
  4. 2025Goku: Flow Based Video Generative Foundation Models
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models
  6. 2023GIVT: Generative Infinite-Vocabulary Transformers
  7. 2025Pushing the Boundaries of State Space Models for Image and Video Generation
  8. 2026VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders