# PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplines

arXiv: https://arxiv.org/abs/2407.08418 · HTML: https://mlsanity.com/paper/2407.08418/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 65 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 30 of 46)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [RoboNet: Large-Scale Multi-Robot Learning](https://mlsanity.com/paper/1910.11215/) (2019)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Earthformer: Exploring Space-Time Transformers for Earth System Forecasting](https://mlsanity.com/paper/2207.05833/) (2022)
- [Self-Supervised Visual Planning with Temporal Skip Connections](https://mlsanity.com/paper/1710.05268/) (2017)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [MaskViT: Masked Visual Pre-Training for Video Prediction](https://mlsanity.com/paper/2206.11894/) (2022)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)

## Built on by (most cited first, 6 of 6)

- [FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model](https://mlsanity.com/paper/2410.13925/) (2024)
- [Towards Dependable Optical Flow: Benchmarks for Real-World Spatiotemporal Corruptions](https://mlsanity.com/paper/2411.14865/) (2024)
- [UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines](https://mlsanity.com/paper/2503.20748/) (2025)
- [Neural Network Modeling of Microstructure Complexity Using Digital Libraries](https://mlsanity.com/paper/2501.18189/) (2025)
- [The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?](https://mlsanity.com/paper/2603.20305/) (2026)
- [ViBR-WM: Visual Bayesian Regression for World Modeling](https://mlsanity.com/paper/2609.33844/) (2026)

## Abstract (arXiv)

In this paper, we introduce PredBench, a benchmark tailored for the holistic evaluation of spatio-temporal prediction networks. Despite significant progress in this field, there remains a lack of a standardized framework for a detailed and comparative analysis of various prediction network architectures. PredBench addresses this gap by conducting large-scale experiments, upholding standardized and appropriate experimental settings, and implementing multi-dimensional evaluations. This benchmark integrates 12 widely adopted methods with 15 diverse datasets across multiple application domains, offering extensive evaluation of contemporary spatio-temporal prediction networks. Through meticulous calibration of prediction settings across various applications, PredBench ensures evaluations relevant to their intended use and enables fair comparisons. Moreover, its multi-dimensional evaluation framework broadens the analysis with a comprehensive set of metrics, providing deep insights into the capabilities of models. The findings from our research offer strategic directions for future developments in the field. Our codebase is available at https://github.com/OpenEarthLab/PredBench.