# OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

arXiv: https://arxiv.org/abs/2306.11249 · HTML: https://mlsanity.com/paper/2306.11249/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 68 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 30 of 42)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction](https://mlsanity.com/paper/1610.00081/) (2016)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Visual Attention Network](https://mlsanity.com/paper/2202.09741/) (2022)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Patches Are All You Need?](https://mlsanity.com/paper/2201.09792/) (2022)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive](https://mlsanity.com/paper/2105.07342/) (2021)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [A Review on Deep Learning Techniques for Video Prediction](https://mlsanity.com/paper/2004.05214/) (2020)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions](https://mlsanity.com/paper/2207.14284/) (2022)
- [USB: A Unified Semi-supervised Learning Benchmark for Classification](https://mlsanity.com/paper/2208.07204/) (2022)
- [Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics](https://mlsanity.com/paper/1811.07490/) (2018)

## Built on by (most cited first, 30 of 54)

- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcasting](https://mlsanity.com/paper/2312.06734/) (2023)
- [VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting](https://mlsanity.com/paper/2403.16536/) (2024)
- [DealMVC: Dual Contrastive Calibration for Multi-view Clustering](https://mlsanity.com/paper/2308.09000/) (2023)
- [Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcasting](https://mlsanity.com/paper/2410.23159/) (2024)
- [Video Representation Learning with Joint-Embedding Predictive Architectures](https://mlsanity.com/paper/2412.10925/) (2024)
- [Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning](https://mlsanity.com/paper/2405.20834/) (2024)
- [Diffusion Transformers as Open-World Spatiotemporal Foundation Models](https://mlsanity.com/paper/2411.12164/) (2024)
- [TimeCHEAT: A Channel Harmony Strategy for Irregularly Sampled Multivariate Time Series Analysis](https://mlsanity.com/paper/2412.12886/) (2024)
- [Deterministic Guidance Diffusion Model for Probabilistic Weather Forecasting](https://mlsanity.com/paper/2312.02819/) (2023)
- [Switch EMA: A Free Lunch for Better Flatness and Sharpness](https://mlsanity.com/paper/2402.09240/) (2024)
- [Autaptic Synaptic Circuit Enhances Spatio-temporal Predictive Learning of Spiking Neural Networks](https://mlsanity.com/paper/2406.00405/) (2024)
- [Triplet Attention Transformer for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2310.18698/) (2023)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [A Comprehensive Graph Pooling Benchmark: Effectiveness, Robustness and Generalizability](https://mlsanity.com/paper/2406.09031/) (2024)
- [Evaluating the Generalization Ability of Spatiotemporal Model in Urban Scenario](https://mlsanity.com/paper/2410.04740/) (2024)
- [TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation](https://mlsanity.com/paper/2412.10275/) (2024)
- [PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplines](https://mlsanity.com/paper/2407.08418/) (2024)
- [Video Prediction Transformers without Recurrence or Convolution](https://mlsanity.com/paper/2410.04733/) (2024)
- [Data-driven Precipitation Nowcasting Using Satellite Imagery](https://mlsanity.com/paper/2412.11480/) (2024)
- [Times2D: Multi-Period Decomposition and Derivative Mapping for General Time Series Forecasting](https://mlsanity.com/paper/2504.00118/) (2025)
- [Does Vector Quantization Fail in Spatio-Temporal Forecasting? Exploring a Differentiable Sparse Soft-Vector Quantization Approach](https://mlsanity.com/paper/2312.03406/) (2023)
- [TraffiDent: A Dataset for Understanding the Interplay Between Traffic Dynamics and Incidents](https://mlsanity.com/paper/2407.11477/) (2024)
- [A general reduced-order neural operator for spatio-temporal predictive learning on complex spatial domains](https://mlsanity.com/paper/2409.05508/) (2024)
- [Timealign: A multi-modal object detection method for time misalignment fusing in autonomous driving](https://mlsanity.com/paper/2412.10033/) (2024)
- [UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines](https://mlsanity.com/paper/2503.20748/) (2025)
- [Spatiotemporal Observer Design for Predictive Learning of High-Dimensional Data](https://mlsanity.com/paper/2402.15284/) (2024)
- [TAFormer: A Unified Target-Aware Transformer for Video and Motion Joint Prediction in Aerial Scenes](https://mlsanity.com/paper/2403.18238/) (2024)
- [TAUDiff: Highly efficient kilometer-scale downscaling using generative diffusion models](https://mlsanity.com/paper/2412.13627/) (2024)
- [ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inference](https://mlsanity.com/paper/2509.03565/) (2025)

## Abstract (arXiv)

Spatio-temporal predictive learning is a learning paradigm that enables models to learn spatial and temporal patterns by predicting future frames from given past frames in an unsupervised manner. Despite remarkable progress in recent years, a lack of systematic understanding persists due to the diverse settings, complex implementation, and difficult reproducibility. Without standardization, comparisons can be unfair and insights inconclusive. To address this dilemma, we propose OpenSTL, a comprehensive benchmark for spatio-temporal predictive learning that categorizes prevalent approaches into recurrent-based and recurrent-free models. OpenSTL provides a modular and extensible framework implementing various state-of-the-art methods. We conduct standard evaluations on datasets across various domains, including synthetic moving object trajectory, human motion, driving scenes, traffic flow and weather forecasting. Based on our observations, we provide a detailed analysis of how model architecture and dataset properties affect spatio-temporal predictive learning performance. Surprisingly, we find that recurrent-free models achieve a good balance between efficiency and performance than recurrent models. Thus, we further extend the common MetaFormers to boost recurrent-free spatial-temporal predictive learning. We open-source the code and models at https://github.com/chengtan9907/OpenSTL.