mlsanityData to 7 Oct 2026

Predicting Video with VQVAE

arXiv 2103.01950 ↗First posted 2 Mar 2021cs.CV cs.LGcited by 61

Not on the map. Why?

Abstract (arXiv)

In recent years, the task of video prediction-forecasting future video given past video frames-has attracted attention in the research community. In this paper we propose a novel approach to this problem with Vector Quantized Variational AutoEncoders (VQ-VAE). With VQ-VAE we compress high-resolution videos into a hierarchical set of multi-scale discrete latent variables. Compared to pixels, this compressed latent space has dramatically reduced dimensionality, allowing us to apply scalable autoregressive generative models to predict video. In contrast to previous work that has largely emphasized highly constrained datasets, we focus on very diverse, large-scale datasets such as Kinetics-600. We predict video at a higher resolution on unconstrained videos, 256x256, than any other previous method to our knowledge. We further validate our approach against prior work via a crowdsourced human evaluation.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

51 of 60 references.

References · 51

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2018World Modelscited by 1,444
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  9. 2016Pixel Recurrent Neural Networkscited by 814
  10. 2020Jukebox: A Generative Model for Musiccited by 535
  11. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  12. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
38 more
  1. 2018MesoNet: a Compact Facial Video Forgery Detection Networkcited by 409
  2. 2016Generating Videos with Scene Dynamicscited by 376
  3. 2018A Short Note about Kinetics-600cited by 331
  4. 2016Dynamic Filter Networkscited by 318
  5. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  6. 2015A note on the evaluation of generative modelscited by 291
  7. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  8. 2018Stochastic Video Generation with a Learned Priorcited by 244
  9. 2017Stochastic Variational Video Predictioncited by 234
  10. 2019Scaling Autoregressive Video Modelscited by 189
  11. 2018Stochastic Adversarial Video Predictioncited by 173
  12. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  13. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  14. 2017Video Generation From Textcited by 153
  15. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  16. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  17. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  18. 2018Fighting Fake News: Image Splice Detection via Learned Self-Consistencycited by 136
  19. 2019Recurrent Convolutional Strategies for Face Manipulation Detection in Videoscited by 118
  20. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  21. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  22. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  23. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 88
  24. 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 86
  25. 2018Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscalingcited by 74
  26. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  27. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  28. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66
  29. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  30. 2018Flow-Grounded Spatial-Temporal Video Prediction from Still Imagescited by 54
  31. 2017Predicting Deeper into the Future of Semantic Segmentationcited by 51
  32. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  33. 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
  34. 2018Disentangling Propagation and Generation for Video Predictioncited by 44
  35. 2018Predicting Future Instance Segmentation by Forecasting Convolutional Featurescited by 37
  36. 2019On the generalization of GAN image forensicscited by 36
  37. 2015Spatio-temporal video autoencoder with differentiable memorycited by 32
  38. 2018Video Prediction with Appearance and Motion Conditionscited by 18

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 21; 2023: 12; 2024: 8; 2025: 6; 2026: 2.

61 papers.

Sort:

By area:

  1. 2022Video Diffusion Modelscited by 2,280
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  4. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  5. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  6. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  7. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  8. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  9. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 117
  10. 2022Generating Long Videos of Dynamic Scenescited by 113
  11. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  12. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
38 more
  1. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  2. 2021VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learningcited by 56
  3. 2022Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Imagecited by 55
  4. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  5. 2022Temporally Consistent Transformers for Video Generationcited by 43
  6. 2023Disentanglement via Latent Quantizationcited by 40
  7. 2024Autoregressive Models in Vision: A Surveycited by 39
  8. 2022Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioningcited by 37
  9. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  10. 2025Taming Teacher Forcing for Masked Autoregressive Video Generationcited by 29
  11. 2025Next Block Prediction: Video Generation via Semi-Autoregressive Modelingcited by 26
  12. 2021Transflower: probabilistic autoregressive dance generation with multimodal attentioncited by 22
  13. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  14. 2023LaMD: Latent Motion Diffusion for Image-Conditional Video Generationcited by 21
  15. 2022Unsupervised Image Representation Learning with Deep Latent Particlescited by 19
  16. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 18
  17. 20223D-Aware Video Generationcited by 17
  18. 2023Neural Vector Fields: Implicit Representation by Explicit Learningcited by 16
  19. 2023Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Modelcited by 16
  20. 2021NWT: Towards natural audio-to-video generation with representation learningcited by 14
  21. 2023CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomycited by 13
  22. 2023Sign Language Production with Latent Motion Transformercited by 13
  23. 2022Towards Smooth Video Compositioncited by 12
  24. 2022JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAEcited by 11
  25. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  26. 2021M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformerscited by 8
  27. 2022Compressed Vision for Efficient Video Understandingcited by 8
  28. 2023How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomycited by 8
  29. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  30. 2021Pragmatic Image Compression for Human-in-the-Loop Decision-Makingcited by 7
  31. 2022INR-V: A Continuous Representation Space for Video-based Generative Taskscited by 7
  32. 2023Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformerscited by 7
  33. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  34. 2021Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Modelscited by 6
  35. 2022SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketchescited by 6
  36. 2021A model of semantic completion in generative episodic memorycited by 5
  37. 2026OmniGen-AR: AutoRegressive Any-to-Image Generationcited by 4
  38. 2021KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrogramscited by 3

50 most cited of 61 All 61 (JSON)