# PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

arXiv: https://arxiv.org/abs/1804.06300 · HTML: https://mlsanity.com/paper/1804.06300/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2018.

## Built on by (most cited first, 30 of 122)

- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2206.12126/) (2022)
- [Latency-Aware Collaborative Perception](https://mlsanity.com/paper/2207.08560/) (2022)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Efficient Video Prediction via Sparsely Conditioned Flow Matching](https://mlsanity.com/paper/2211.14575/) (2022)
- [OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning](https://mlsanity.com/paper/2306.11249/) (2023)
- [DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos](https://mlsanity.com/paper/2304.06544/) (2023)
- [VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting](https://mlsanity.com/paper/2403.16536/) (2024)
- [SwinVRNN: A Data-Driven Ensemble Forecasting Model via Learned Distribution Perturbation](https://mlsanity.com/paper/2205.13158/) (2022)
- [SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM](https://mlsanity.com/paper/2308.09891/) (2023)
- [STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Prediction](https://mlsanity.com/paper/2203.16084/) (2022)
- [SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2211.12509/) (2022)
- [MMVP: Motion-Matrix-based Video Prediction](https://mlsanity.com/paper/2308.16154/) (2023)
- [VPTR: Efficient Transformers for Video Prediction](https://mlsanity.com/paper/2203.15836/) (2022)
- [Dynamics-Aware Spatiotemporal Occupancy Prediction in Urban Environments](https://mlsanity.com/paper/2209.13172/) (2022)
- [MIMO Is All You Need : A Strong Multi-In-Multi-Out Baseline for Video Prediction](https://mlsanity.com/paper/2212.04655/) (2022)
- [Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcasting](https://mlsanity.com/paper/2410.23159/) (2024)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [Advancing Learned Video Compression with In-loop Frame Prediction](https://mlsanity.com/paper/2211.07004/) (2022)
- [Implicit Stacked Autoregressive Model for Video Prediction](https://mlsanity.com/paper/2303.07849/) (2023)
- [Video Prediction by Efficient Transformers](https://mlsanity.com/paper/2212.06026/) (2022)
- [TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Auto-Regression](https://mlsanity.com/paper/2304.14131/) (2023)
- [Diffusion Transformers as Open-World Spatiotemporal Foundation Models](https://mlsanity.com/paper/2411.12164/) (2024)
- [Local Frequency Domain Transformer Networks for Video Prediction](https://mlsanity.com/paper/2105.04637/) (2021)
- [Learning Scene Dynamics from Point Cloud Sequences](https://mlsanity.com/paper/2111.08755/) (2021)
- [Deterministic Guidance Diffusion Model for Probabilistic Weather Forecasting](https://mlsanity.com/paper/2312.02819/) (2023)
- [GPTCast: a weather language model for precipitation nowcasting](https://mlsanity.com/paper/2407.02089/) (2024)
- [Physics-informed generative neural network: an application to troposphere temperature prediction](https://mlsanity.com/paper/2107.06991/) (2021)
- [Learning to Transfer for Traffic Forecasting via Multi-task Learning](https://mlsanity.com/paper/2111.15542/) (2021)

## Abstract (arXiv)

We present PredRNN++, an improved recurrent network for video predictive learning. In pursuit of a greater spatiotemporal modeling capability, our approach increases the transition depth between adjacent states by leveraging a novel recurrent unit, which is named Causal LSTM for re-organizing the spatial and temporal memories in a cascaded mechanism. However, there is still a dilemma in video predictive learning: increasingly deep-in-time models have been designed for capturing complex variations, while introducing more difficulties in the gradient back-propagation. To alleviate this undesirable effect, we propose a Gradient Highway architecture, which provides alternative shorter routes for gradient flows from outputs back to long-range inputs. This architecture works seamlessly with causal LSTMs, enabling PredRNN++ to capture short-term and long-term dependencies adaptively. We assess our model on both synthetic and real video datasets, showing its ability to ease the vanishing gradient problem and yield state-of-the-art prediction results even in a difficult objects occlusion scenario.