mlsanityData to 7 Oct 2026

PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning

arXiv 2103.09504 ↗First posted 17 Mar 2021cs.LG cs.CVcited by 181

Not on the map. Why?

Abstract (arXiv)

The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical context, where the visual dynamics are believed to have modular structures that can be learned with compositional subsystems. This paper models these structures by presenting PredRNN, a new recurrent network, in which a pair of memory cells are explicitly decoupled, operate in nearly independent transition manners, and finally form unified representations of the complex environment. Concretely, besides the original memory cell of LSTM, this network is featured by a zigzag memory flow that propagates in both bottom-up and top-down directions across all layers, enabling the learned visual dynamics at different levels of RNNs to communicate. It also leverages a memory decoupling loss to keep the memory cells from learning redundant features. We further propose a new curriculum learning strategy to force PredRNN to learn long-term dynamics from context frames, which can be generalized to most sequence-to-sequence models. We provide detailed ablation studies to verify the effectiveness of each component. Our approach is shown to obtain highly competitive results on five datasets for both action-free and action-conditioned predictive learning scenarios.

Key facts. Cited by 181 papers in the map (data through 7 Oct 2026). Builds on 61 of its 96 references that are in the map. First posted on arXiv 17 Mar 2021.

Builds on

61 of 96 references.

References · 61

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  7. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  10. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  12. 2014Neural Turing Machinescited by 630
38 more
  1. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  2. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  3. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  4. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  5. 2018Neural Relational Inference for Interacting Systemscited by 339
  6. 2016Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Predictioncited by 330
  7. 2016Dynamic Filter Networkscited by 318
  8. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  9. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  10. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  11. 2018Stochastic Video Generation with a Learned Priorcited by 244
  12. 2017Stochastic Variational Video Predictioncited by 234
  13. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  14. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  15. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  16. 2020Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Predictioncited by 192
  17. 2019Scaling Autoregressive Video Modelscited by 189
  18. 2017Deep Learning for Precipitation Nowcasting: A Benchmark and A New Modelcited by 180
  19. 2018Stochastic Adversarial Video Predictioncited by 173
  20. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  21. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  22. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  23. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  24. 2018PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learningcited by 122
  25. 2018Learning to Decompose and Disentangle Representations for Video Predictioncited by 120
  26. 2020A Review on Deep Learning Techniques for Video Predictioncited by 114
  27. 2016Video Pixel Networkscited by 108
  28. 2020Stochastic Latent Residual Video Predictioncited by 106
  29. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  30. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  31. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  32. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  33. 2018Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamicscited by 91
  34. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  35. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  36. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  37. 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
  38. 2017Recurrent Environment Simulatorscited by 68

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 21; 2023: 35; 2024: 46; 2025: 37; 2026: 36.

181 papers.

Sort:

By area:

  1. 2022SimVP: Simpler yet Better Video Predictioncited by 181
  2. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  3. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  4. 2023PreDiff: Precipitation Nowcasting with Latent Diffusion Modelscited by 96
  5. 2022Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learningcited by 84
  6. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  7. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 53
  8. 2023PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Predictioncited by 44
  9. 2024VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecastingcited by 35
  10. 2023Foundation Models for Weather and Climate Data Understanding: A Comprehensive Surveycited by 32
  11. 2025OneForecast: A Universal Framework for Global and Regional Weather Forecastingcited by 30
  12. 2024Networking Systems for Video Anomaly Detection: A Tutorial and Surveycited by 21
38 more
  1. 2022SimVPv2: Towards Simple yet Powerful Spatiotemporal Predictive Learningcited by 20
  2. 2023Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactionscited by 19
  3. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 18
  4. 2023MMVP: Motion-Matrix-based Video Predictioncited by 18
  5. 2023Multi-modal learning for geospatial vegetation forecastingcited by 16
  6. 2023Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Modelcited by 16
  7. 2024Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcastingcited by 16
  8. 2025Deep Learning and Foundation Models for Weather Prediction: A Surveycited by 15
  9. 2023Implicit Stacked Autoregressive Model for Video Predictioncited by 14
  10. 2024Deep learning for precipitation nowcasting: A survey from the perspective of time series forecastingcited by 14
  11. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  12. 2026Walk through Paintings: Egocentric World Models from Internet Priorscited by 13
  13. 2022MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networkscited by 12
  14. 2022Neighbor Correspondence Matching for Flow-based Video Frame Synthesiscited by 12
  15. 2023TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Auto-Regressioncited by 12
  16. 2022On the Learning Mechanisms in Physical Reasoningcited by 11
  17. 2023Physics-aware Machine Learning Revolutionizes Scientific Paradigm for Machine Learning and Process-based Hydrologycited by 11
  18. 2024E-Motion: Future Motion Simulation via Event Sequence Diffusioncited by 11
  19. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  20. 2024PostCast: Generalizable Postprocessing for Precipitation Nowcasting via Unsupervised Blurriness Modelingcited by 10
  21. 2025Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Modelscited by 8
  22. 2022Predicting Future Occupancy Grids in Dynamic Environment with Spatio-Temporal Learningcited by 7
  23. 2023Triplet Attention Transformer for Spatiotemporal Predictive Learningcited by 7
  24. 2024Self-supervised Multi-future Occupancy Forecasting for Autonomous Drivingcited by 7
  25. 2022Data-Driven Short-Term Daily Operational Sea Ice Regional Forecastingcited by 6
  26. 2022WeatherFusionNet: Predicting Precipitation from Satellite Datacited by 6
  27. 2024PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplinescited by 6
  28. 2024Video Prediction Transformers without Recurrence or Convolutioncited by 6
  29. 2024Data-driven Precipitation Nowcasting Using Satellite Imagerycited by 6
  30. 2025FlowCast: Advancing Precipitation Nowcasting with Conditional Flow Matchingcited by 6
  31. 2021Interpretable Deep Feature Propagation for Early Action Recognitioncited by 5
  32. 2022MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learningcited by 5
  33. 2022Exploiting Spatial-temporal Correlations for Video Anomaly Detectioncited by 5
  34. 2022LGN-Net: Local-Global Normality Network for Video Anomaly Detectioncited by 5
  35. 2023S-HR-VQVAE: Sequential Hierarchical Residual Learning Vector Quantized Variational Autoencoder for Video Predictioncited by 5
  36. 2023AdapTraj: A Multi-Source Domain Generalization Framework for Multi-Agent Trajectory Predictioncited by 5
  37. 2024Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidancecited by 5
  38. 2025High-Dynamic Radar Sequence Prediction for Weather Nowcasting Using Spatiotemporal Coherent Gaussian Representationcited by 5

50 most cited of 181 All 181 (JSON)