mlsanityData to 7 Oct 2026

Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction

arXiv 2103.04174 ↗First posted 6 Mar 2021cs.CV cs.AI cs.LGcited by 75

Not on the map. Why?

Abstract (arXiv)

A video prediction model that generalizes to diverse scenes would enable intelligent agents such as robots to perform a variety of tasks via planning with the model. However, while existing video prediction models have produced promising results on small datasets, they suffer from severe underfitting when trained on large and diverse datasets. To address this underfitting challenge, we first observe that the ability to train larger video prediction models is often bottlenecked by the memory constraints of GPUs or TPUs. In parallel, deep hierarchical latent variable models can produce higher quality predictions by capturing the multi-level stochasticity of future observations, but end-to-end optimization of such models is notably difficult. Our key insight is that greedy and modular optimization of hierarchical autoencoders can simultaneously address both the memory constraints and the optimization challenges of large-scale video prediction. We introduce Greedy Hierarchical Variational Autoencoders (GHVAEs), a method that learns high-fidelity video predictions by greedily training each level of a hierarchical autoencoder. In comparison to state-of-the-art models, GHVAEs provide 17-55% gains in prediction performance on four video datasets, a 35-40% higher success rate on real robot tasks, and can improve performance monotonically by simply adding more modules.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 60 of its 87 references that are in the map. First posted on arXiv 6 Mar 2021.

Builds on

60 of 87 references.

References · 60

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  9. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  10. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  11. 2016Dynamic Filter Networkscited by 318
  12. 2016Ladder Variational Autoencoderscited by 315
38 more
  1. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  3. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  4. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  5. 2018Stochastic Video Generation with a Learned Priorcited by 244
  6. 2017Stochastic Variational Video Predictioncited by 234
  7. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  8. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  9. 2017Video Frame Synthesis using Deep Voxel Flowcited by 209
  10. 2019Contrastive Learning of Structured World Modelscited by 206
  11. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  12. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  13. 2018Stochastic Adversarial Video Predictioncited by 173
  14. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  15. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  16. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  17. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  18. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  19. 2019BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modelingcited by 109
  20. 2016Video Pixel Networkscited by 108
  21. 2020Stochastic Latent Residual Video Predictioncited by 106
  22. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  23. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  24. 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 86
  25. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  26. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  27. 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
  28. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
  29. 2017Towards Deeper Understanding of Variational Autoencoding Modelscited by 56
  30. 2019COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Explorationcited by 55
  31. 2019Deep Visual MPC-Policy Learning for Navigationcited by 54
  32. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  33. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  34. 2018SDCNet: Video Prediction Using Spatially-Displaced Convolutioncited by 50
  35. 2016SE3-Nets: Learning Rigid Body Motion using Deep Neural Networkscited by 48
  36. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  37. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  38. 2019Compositional Video Predictioncited by 43

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 27; 2023: 20; 2024: 8; 2025: 8; 2026: 3.

75 papers.

Sort:

By area:

  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  2. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  3. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  4. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  5. 2021PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learningcited by 181
  6. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  7. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  8. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  9. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  10. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  11. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  12. 2021GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinementcited by 109
38 more
  1. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  2. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  3. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  4. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  5. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  6. 2022Do better ImageNet classifiers assess perceptual similarity better?cited by 34
  7. 2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
  8. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  9. 2021Sawtooth Factorial Topic Embeddings Guided Gamma Belief Networkcited by 23
  10. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  11. 2023Blockwise Self-Supervised Learning at Scalecited by 22
  12. 2023PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videoscited by 22
  13. 2023Slot-VAE: Object-Centric Scene Generation with Slot Attentioncited by 20
  14. 2022Unsupervised Image Representation Learning with Deep Latent Particlescited by 19
  15. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 18
  16. 2022STAP: Sequencing Task-Agnostic Policiescited by 17
  17. 2025World Models That Know When They Don't Know - Controllable Video Generation with Calibrated Uncertaintycited by 16
  18. 2023Video Diffusion Models with Local-Global Context Guidancecited by 15
  19. 2023Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenescited by 13
  20. 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 12
  21. 2022MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networkscited by 12
  22. 2023MOSO: Decomposing MOtion, Scene and Object for Video Predictioncited by 12
  23. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  24. 2023STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Predictioncited by 12
  25. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  26. 2023Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspectivecited by 10
  27. 2021Variational Predictive Routing with Nested Subjective Timescalescited by 9
  28. 2024Momentum Auxiliary Network for Supervised Local Learningcited by 9
  29. 2025STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flowscited by 9
  30. 2022Continual Predictive Learning from Videoscited by 8
  31. 2024Scaling Supervised Local Learning with Augmented Auxiliary Networkscited by 8
  32. 2025Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Modelscited by 8
  33. 2022A unified model for continuous conditional video predictioncited by 7
  34. 2023Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformerscited by 7
  35. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  36. 2023Block-local learning with probabilistic latent representationscited by 6
  37. 2024PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplinescited by 6
  38. 2022MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learningcited by 5

50 most cited of 75 All 75 (JSON)