Long-Term Prediction of Natural Video Sequences with Robust Video Predictors
Not on the map. Why?
Abstract (arXiv)
Predicting high dimensional video sequences is a curiously difficult problem. The number of possible futures for a given video sequence grows exponentially over time due to uncertainty. This is especially evident when trying to predict complicated natural video scenes from a limited snapshot of the world. The inherent uncertainty accumulates the further into the future you predict making long-term prediction very difficult. In this work we introduce a number of improvements to existing work that aid in creating Robust Video Predictors (RoViPs). We show that with a combination of deep Perceptual and uncertainty-based reconstruction losses we are able to create high quality short-term predictions. Attention-based skip connections are utilised to allow for long range spatial movement of input features to further improve performance. Finally, we show that by simply making the predictor robust to its own prediction errors, it is possible to produce very long, realistic natural video sequences using an iterated single-step prediction task.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 21 Aug 2023.
Builds on
24 of 38 references.
References · 24
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2017Non-local Neural Networkscited by 2,290
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
- 2018Self-Attention Generative Adversarial Networkscited by 809
- 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
- 2015Deep multi-scale video prediction beyond mean square errorcited by 297
- 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
- 2017Stochastic Variational Video Predictioncited by 234
12 more
- 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
- 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
- 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
- 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
- 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
- 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
- 2020Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Predictioncited by 52
- 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
- 2018SDCNet: Video Prediction Using Spatially-Displaced Convolutioncited by 50
- 2018Bayesian Prediction of Future Street Scenes using Synthetic Likelihoodscited by 17
- 2015Unsupervised Learning of Visual Structure using Predictive Generative Networkscited by 12
- 2020Photo-Realistic Video Prediction on Natural Videos of Largely Changing Framescited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.