mlsanityData to 13 Oct 2026

Stochastic Latent Residual Video Prediction

arXiv 2002.09219 ↗First posted 21 Feb 2020cs.CV cs.LG stat.MLcited by 108

Not on the map. Why?

Abstract (arXiv)

Designing video prediction models that account for the inherent uncertainty of the future is challenging. Most works in the literature are based on stochastic image-autoregressive recurrent networks, which raises several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and temporal dynamics. However, no such model for stochastic video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model whose dynamics are governed in a latent space by a residual update rule. This first-order scheme is motivated by discretization schemes of differential equations. It naturally models video dynamics as it allows our simpler, more interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

Key facts. Cited by 108 papers in the map (data through 13 Oct 2026). First posted on arXiv 21 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 30; 2022: 28; 2023: 24; 2024: 15; 2025: 8; 2026: 3.

108 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,550
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,467
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 627
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 466
  5. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 311
  6. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 308
  7. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 260
  8. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  9. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  10. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  11. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  12. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
38 more
  1. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  2. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  3. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  4. 2023Generative Image Dynamicscited by 94
  5. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  6. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  7. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  8. 2021Playable Video Generationcited by 64
  9. 2023Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Treescited by 60
  10. 2023DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcastingcited by 58
  11. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 55
  12. 2024Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videoscited by 55
  13. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  14. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  15. 2021Stochastic Image-to-Video Synthesis using cINNscited by 49
  16. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  17. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  18. 2021Controllable Animation of Fluid Elements in Still Imagescited by 44
  19. 2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
  20. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  21. 2021Clockwork Variational Autoencoderscited by 41
  22. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
  23. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  24. 2022Deep Latent State Space Models for Time-Series Generationcited by 37
  25. 2021EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction taskcited by 34
  26. 2022Do better ImageNet classifiers assess perceptual similarity better?cited by 34
  27. 2021SLAMP: Stochastic Latent Appearance and Motion Predictioncited by 32
  28. 2023UniWorld: Autonomous Driving Pre-training via World Modelscited by 32
  29. 2022STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Predictioncited by 31
  30. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  31. 2025Object-Centric World Model for Language-Guided Manipulationcited by 16
  32. 2023Video Diffusion Models with Local-Global Context Guidancecited by 15
  33. 2023Implicit Stacked Autoregressive Model for Video Predictioncited by 14
  34. 2025MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generationcited by 14
  35. 2021Accurate Grid Keypoint Learning for Efficient Video Predictioncited by 13
  36. 2021Diverse Generation from a Single Video Made Possiblecited by 13
  37. 2022Video Prediction by Efficient Transformerscited by 12
  38. 2021Click to Move: Controlling Video Generation with Sparse Motioncited by 11

50 most cited of 108 All 108 (JSON)