mlsanityData to 14 Oct 2026

Improved Conditional VRNNs for Video Prediction

arXiv 1904.12165 ↗First posted 27 Apr 2019cs.CV cs.LGcited by 104

Not on the map. Why?

Abstract (arXiv)

Predicting future frames for a video sequence is a challenging generative modeling task. Promising approaches include probabilistic latent variable models such as the Variational Auto-Encoder. While VAEs can handle uncertainty and model multiple possible future outcomes, they have a tendency to produce blurry predictions. In this work we argue that this is a sign of underfitting. To address this issue, we propose to increase the expressiveness of the latent distributions and to use higher capacity likelihood models. Our approach relies on a hierarchy of latent variables, which defines a family of flexible prior and posterior distributions in order to better model the probability of future sequences. We validate our proposal through a series of ablation experiments and compare our approach to current state-of-the-art latent variable models. Our method performs favorably under several metrics in three different datasets.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 29; 2022: 26; 2023: 27; 2024: 13; 2025: 7; 2026: 2.

104 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  3. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  4. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  5. 2022SimVP: Simpler yet Better Video Predictioncited by 184
  6. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 182
  7. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  8. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 103
  9. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  10. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  11. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  12. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 86
38 more
  1. 2023LLM-grounded Video Diffusion Modelscited by 83
  2. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  3. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  4. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  5. 2023Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Treescited by 62
  6. 2024Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videoscited by 58
  7. 2022Efficient Video Prediction via Sparsely Conditioned Flow Matchingcited by 55
  8. 2021MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motionscited by 54
  9. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 54
  10. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  11. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  12. 2024On the Content Bias in Fr'echet Video Distancecited by 52
  13. 2021Learning the Predictability of the Futurecited by 49
  14. 2021Stochastic Image-to-Video Synthesis using cINNscited by 49
  15. 2023Breathing Life Into Sketches Using Text-to-Video Priorscited by 48
  16. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  17. 2021Controllable Animation of Fluid Elements in Still Imagescited by 45
  18. 2024DINO-Foresight: Looking into the Future with DINOcited by 45
  19. 2021Clockwork Variational Autoencoderscited by 42
  20. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  21. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  22. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  23. 2021SLAMP: Stochastic Latent Appearance and Motion Predictioncited by 32
  24. 2023Explicitly Minimizing the Blur Error of Variational Autoencoderscited by 31
  25. 2021Revisiting Hierarchical Approach for Persistent Long-Term Video Predictioncited by 23
  26. 2022SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learningcited by 22
  27. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  28. 2021Diverse Video Generation using a Gaussian Process Triggercited by 17
  29. 2022VPTR: Efficient Transformers for Video Predictioncited by 17
  30. 2021Comparing Correspondences: Video Prediction with Correspondence-wise Lossescited by 16
  31. 2025Beyond the Frame: Generating 360 Panoramic Videos from Perspective Videoscited by 16
  32. 2023Video Diffusion Models with Local-Global Context Guidancecited by 15
  33. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetcited by 15
  34. 2023Implicit Stacked Autoregressive Model for Video Predictioncited by 14
  35. 2021Accurate Grid Keypoint Learning for Efficient Video Predictioncited by 13
  36. 2022MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networkscited by 13
  37. 2023Automatic Animation of Hair Blowing in Still Portrait Photoscited by 13
  38. 2022Video Prediction by Efficient Transformerscited by 12

50 most cited of 104 All 104 (JSON)