# STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Prediction

arXiv: https://arxiv.org/abs/2203.16084 · HTML: https://mlsanity.com/paper/2203.16084/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Video (language) modeling: a baseline for generative models of natural videos](https://mlsanity.com/paper/1412.6604/) (2014)
- [High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks](https://mlsanity.com/paper/1911.01655/) (2019)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty](https://mlsanity.com/paper/1711.09026/) (2017)
- [PhaseNet for Video Frame Interpolation](https://mlsanity.com/paper/1804.00884/) (2018)
- [MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions](https://mlsanity.com/paper/2103.02243/) (2021)
- [Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction](https://mlsanity.com/paper/2002.09905/) (2020)
- [Video Prediction via Example Guidance](https://mlsanity.com/paper/2007.01738/) (2020)

## Built on by (most cited first, 30 of 31)

- [DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcasting](https://mlsanity.com/paper/2312.06734/) (2023)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction](https://mlsanity.com/paper/2305.11421/) (2023)
- [Multi-view Aggregation Network for Dichotomous Image Segmentation](https://mlsanity.com/paper/2404.07445/) (2024)
- [SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2211.12509/) (2022)
- [MMVP: Motion-Matrix-based Video Prediction](https://mlsanity.com/paper/2308.16154/) (2023)
- [Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model](https://mlsanity.com/paper/2312.08403/) (2023)
- [Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcasting](https://mlsanity.com/paper/2410.23159/) (2024)
- [Implicit Stacked Autoregressive Model for Video Prediction](https://mlsanity.com/paper/2303.07849/) (2023)
- [E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning](https://mlsanity.com/paper/2311.18433/) (2023)
- [Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey](https://mlsanity.com/paper/2405.01725/) (2024)
- [MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.03010/) (2022)
- [WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction](https://mlsanity.com/paper/2211.14308/) (2022)
- [S-HR-VQVAE: Sequential Hierarchical Residual Learning Vector Quantized Variational Autoencoder for Video Prediction](https://mlsanity.com/paper/2307.06701/) (2023)
- [MS-LSTM: Exploring Spatiotemporal Multiscale Representations in Video Prediction Domain](https://mlsanity.com/paper/2304.07724/) (2023)
- [Motion Graph Unleashed: A Novel Approach to Video Prediction](https://mlsanity.com/paper/2410.22288/) (2024)
- [Weakly-supervised Representation Learning for Video Alignment and Analysis](https://mlsanity.com/paper/2302.04064/) (2023)
- [STIP: A SpatioTemporal Information-Preserving and Perception-Augmented Model for High-Resolution Video Prediction](https://mlsanity.com/paper/2206.04381/) (2022)
- [Predictive Coding Based Multiscale Network with Encoder-Decoder LSTM for Video Prediction](https://mlsanity.com/paper/2212.11642/) (2022)
- [RainPro-8: An Efficient Deep Learning Model to Estimate Rainfall Probabilities Over 8 Hours](https://mlsanity.com/paper/2505.10271/) (2025)
- [Stable Attention Response for Reliable Precipitation Nowcasting](https://mlsanity.com/paper/2605.13181/) (2026)
- [McCast: Memory-Guided Latent Drift Correction for Long-Horizon Precipitation Nowcasting](https://mlsanity.com/paper/2605.13197/) (2026)
- [Fast Fourier Inception Networks for Occluded Video Prediction](https://mlsanity.com/paper/2306.10346/) (2023)
- [Masked Autoregressive Model for Weather Forecasting](https://mlsanity.com/paper/2409.20117/) (2024)
- [STLDM: Spatio-Temporal Latent Diffusion Model for Precipitation Nowcasting](https://mlsanity.com/paper/2512.21118/) (2025)
- [Anti-aliasing Predictive Coding Network for Future Video Frame Prediction](https://mlsanity.com/paper/2301.05421/) (2023)
- [Pair-wise Layer Attention with Spatial Masking for Video Prediction](https://mlsanity.com/paper/2311.11289/) (2023)
- [SIAM: A Simple Alternating Mixer for Video Prediction](https://mlsanity.com/paper/2311.11683/) (2023)
- [State-space Decomposition Model for Video Prediction Considering Long-term Motion Trend](https://mlsanity.com/paper/2404.11576/) (2024)
- [Data-Driven Uncertainty-Aware Forecasting of Sea Ice Conditions in the Gulf of Ob Based on Satellite Radar Imagery](https://mlsanity.com/paper/2410.19782/) (2024)

## Abstract (arXiv)

Although many video prediction methods have obtained good performance in low-resolution (64~128) videos, predictive models for high-resolution (512~4K) videos have not been fully explored yet, which are more meaningful due to the increasing demand for high-quality videos. Compared with low-resolution videos, high-resolution videos contain richer appearance (spatial) information and more complex motion (temporal) information. In this paper, we propose a Spatiotemporal Residual Predictive Model (STRPM) for high-resolution video prediction. On the one hand, we propose a Spatiotemporal Encoding-Decoding Scheme to preserve more spatiotemporal information for high-resolution videos. In this way, the appearance details for each frame can be greatly preserved. On the other hand, we design a Residual Predictive Memory (RPM) which focuses on modeling the spatiotemporal residual features (STRF) between previous and future frames instead of the whole frame, which can greatly help capture the complex motion information in high-resolution videos. In addition, the proposed RPM can supervise the spatial encoder and temporal encoder to extract different features in the spatial domain and the temporal domain, respectively. Moreover, the proposed model is trained using generative adversarial networks (GANs) with a learned perceptual loss (LP-loss) to improve the perceptual quality of the predictions. Experimental results show that STRPM can generate more satisfactory results compared with various existing methods.