# Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames

arXiv: https://arxiv.org/abs/2003.08635 · HTML: https://mlsanity.com/paper/2003.08635/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 8 of 8)

- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.12126/) (2022)
- [SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2211.12509/) (2022)
- [Implicit Stacked Autoregressive Model for Video Prediction](https://mlsanity.com/paper/2303.07849/) (2023)
- [RAP-Net: Region Attention Predictive Network for Precipitation Nowcasting](https://mlsanity.com/paper/2110.01035/) (2021)
- [Long-Term Prediction of Natural Video Sequences with Robust Video Predictors](https://mlsanity.com/paper/2308.11079/) (2023)
- [Towards an Interpretable Latent Space in Structured Models for Video Prediction](https://mlsanity.com/paper/2107.07713/) (2021)
- [Masked Autoregressive Model for Weather Forecasting](https://mlsanity.com/paper/2409.20117/) (2024)

## Abstract (arXiv)

Recent advances in deep learning have significantly improved performance of video prediction. However, state-of-the-art methods still suffer from blurriness and distortions in their future predictions, especially when there are large motions between frames. To address these issues, we propose a deep residual network with the hierarchical architecture where each layer makes a prediction of future state at different spatial resolution, and these predictions of different layers are merged via top-down connections to generate future frames. We trained our model with adversarial and perceptual loss functions, and evaluated it on a natural video dataset captured by car-mounted cameras. Our model quantitatively outperforms state-of-the-art baselines in future frame prediction on video sequences of both largely and slightly changing frames. Furthermore, our model generates future frames with finer details and textures that are perceptually more realistic than the baselines, especially under fast camera motions.