mlsanityData to 7 Oct 2026

High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks

arXiv 1911.01655 ↗First posted 5 Nov 2019cs.CVcited by 88

Not on the map. Why?

Abstract (arXiv)

Predicting future video frames is extremely challenging, as there are many factors of variation that make up the dynamics of how frames change through time. Previously proposed solutions require complex inductive biases inside network architectures with highly specialized computation, including segmentation masks, optical flow, and foreground and background separation. In this work, we question if such handcrafted architectures are necessary and instead propose a different approach: finding minimal inductive bias for video prediction while maximizing network capacity. We investigate this question by performing the first large-scale empirical study and demonstrate state-of-the-art performance by learning large models on three different datasets: one for modeling object interactions, one for modeling human motion, and one for modeling car driving.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 31; 2023: 11; 2024: 12; 2025: 10; 2026: 3.

88 papers.

Sort:

By area:

  1. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  2. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  3. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  4. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  5. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  6. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  7. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  8. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  9. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  10. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  11. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  12. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
38 more
  1. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  2. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  3. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  4. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  5. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  6. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  7. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 54
  8. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  9. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  10. 2021Controllable Animation of Fluid Elements in Still Imagescited by 44
  11. 2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
  12. 2022Temporally Consistent Transformers for Video Generationcited by 43
  13. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
  14. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  15. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  16. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  17. 2021SLAMP: Stochastic Latent Appearance and Motion Predictioncited by 32
  18. 2022STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Predictioncited by 31
  19. 2022Optimizing Video Prediction via Video Frame Interpolationcited by 28
  20. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  21. 2023A Control-Centric Benchmark for Video Predictioncited by 25
  22. 2021Revisiting Hierarchical Approach for Persistent Long-Term Video Predictioncited by 23
  23. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  24. 2022Unsupervised Image Representation Learning with Deep Latent Particlescited by 19
  25. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 18
  26. 2021Diverse Video Generation using a Gaussian Process Triggercited by 17
  27. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  28. 2021Comparing Correspondences: Video Prediction with Correspondence-wise Lossescited by 16
  29. 2022Variational Causal Dynamics: Discovering Modular World Models from Interventionscited by 16
  30. 2025MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generationcited by 14
  31. 2021Accurate Grid Keypoint Learning for Efficient Video Predictioncited by 13
  32. 2021Diverse Generation from a Single Video Made Possiblecited by 13
  33. 2023Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenescited by 13
  34. 2022MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networkscited by 12
  35. 2023MOSO: Decomposing MOtion, Scene and Object for Video Predictioncited by 12
  36. 2021Future Frame Prediction for Robot-assisted Surgerycited by 11
  37. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  38. 2024SPARTAN: A Sparse Transformer World Model Attending to What Matterscited by 10

50 most cited of 88 All 88 (JSON)