mlsanityData to 7 Oct 2026

Stochastic Variational Video Prediction

arXiv 1710.11252 ↗First posted 30 Oct 2017cs.CV cs.ROcited by 234

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images requires the predictive model to build an intricate understanding of the natural world. Many existing methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this paper, we develop a stochastic variational video prediction (SV2P) method that predicts a different possible future for each sample of its latent variables. To the best of our knowledge, our model is the first to provide effective stochastic multi-frame prediction for real-world video. We demonstrate the capability of the proposed method in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods. Our SV2P implementation will be open sourced upon publication.

Key facts. Cited by 234 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 2; 2021: 56; 2022: 62; 2023: 46; 2024: 34; 2025: 21; 2026: 12.

234 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  6. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  7. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  8. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  9. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  10. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  11. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  12. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
38 more
  1. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  2. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 251
  3. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  4. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  5. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  6. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  7. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  8. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  9. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  10. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  11. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  12. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  13. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  14. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  15. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  16. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  17. 2022Generating Long Videos of Dynamic Scenescited by 113
  18. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  19. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  20. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  21. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  22. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  23. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  24. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  25. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  26. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  27. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  28. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  29. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  30. 2021Playable Video Generationcited by 63
  31. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  32. 2021Predicting Video with VQVAEcited by 61
  33. 2022SinFusion: Training Diffusion Models on a Single Image or Videocited by 58
  34. 2023DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcastingcited by 57
  35. 2025Embodied AI Agents: Modeling the Worldcited by 57
  36. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  37. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  38. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54

50 most cited of 234 All 234 (JSON)