# Spatial-Temporal-Decoupled Masked Pre-training for Spatiotemporal Forecasting

arXiv: https://arxiv.org/abs/2312.00516 · HTML: https://mlsanity.com/paper/2312.00516/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting](https://mlsanity.com/paper/2007.02842/) (2020)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [Spatial-Temporal Fusion Graph Neural Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2012.09641/) (2020)
- [Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2106.12931/) (2021)
- [Spatial-Temporal Transformer Networks for Traffic Flow Forecasting](https://mlsanity.com/paper/2001.02908/) (2020)
- [Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2208.05233/) (2022)
- [PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction](https://mlsanity.com/paper/2301.07945/) (2023)
- [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2206.09113/) (2022)
- [Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis](https://mlsanity.com/paper/2310.06119/) (2023)
- [Spatio-Temporal Meta-Graph Learning for Traffic Forecasting](https://mlsanity.com/paper/2211.14701/) (2022)
- [DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction](https://mlsanity.com/paper/2108.09091/) (2021)
- [Z-GCNETs: Time Zigzags at Graph Convolutional Networks for Time Series Forecasting](https://mlsanity.com/paper/2105.04100/) (2021)
- [Ti-MAE: Self-Supervised Masked Time Series Autoencoders](https://mlsanity.com/paper/2301.08871/) (2023)
- [Towards Spatio-Temporal Aware Traffic Time Series Forecasting--Full Version](https://mlsanity.com/paper/2203.15737/) (2022)
- [A Multi-view Multi-task Learning Framework for Multi-variate Time Series Forecasting](https://mlsanity.com/paper/2109.01657/) (2021)

## Built on by (most cited first, 27 of 27)

- [Message-Passing State-Space Models: Improving Graph Learning with Modern Sequence Modeling](https://mlsanity.com/paper/2505.18728/) (2025)
- [Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review](https://mlsanity.com/paper/2506.01364/) (2025)
- [ST-LoRA: Low-rank Adaptation for Spatio-Temporal Forecasting](https://mlsanity.com/paper/2404.07919/) (2024)
- [Decoupling Spatio-Temporal Prediction: When Lightweight Large Models Meet Adaptive Hypergraphs](https://mlsanity.com/paper/2505.19620/) (2025)
- [How Different from the Past? Spatio-Temporal Time Series Forecasting with Self-Supervised Deviation Learning](https://mlsanity.com/paper/2510.04908/) (2025)
- [Cross Space and Time: A Spatio-Temporal Unitized Model for Traffic Flow Forecasting](https://mlsanity.com/paper/2411.09251/) (2024)
- [Persistent Homology-induced Graph Ensembles for Time Series Regressions](https://mlsanity.com/paper/2503.14240/) (2025)
- [A General ReLearner: Empowering Spatiotemporal Prediction by Re-learning Input-label Residual](https://mlsanity.com/paper/2602.02563/) (2026)
- [UrbanFM: Scaling Urban Spatio-Temporal Foundation Models](https://mlsanity.com/paper/2602.20677/) (2026)
- [Dynamic Sheaf Diffusion Networks with Adaptive Local Structure for Heterogeneous Spatio-Temporal Graph Learning](https://mlsanity.com/paper/2604.11275/) (2026)
- [GAMMA-Net: Adaptive Long-Horizon Traffic Spatio-Temporal Forecasting Model based on Interleaved Graph Attention and Multi-Axis Mamba](https://mlsanity.com/paper/2604.16859/) (2026)
- [Net-Ev2: A Generative Simulator for Network Event Evolution](https://mlsanity.com/paper/2606.12494/) (2026)
- [HiMoE: Heterogeneity-Informed Mixture-of-Experts for Fair Spatial-Temporal Forecasting](https://mlsanity.com/paper/2412.00316/) (2024)
- [Embedding spatial context in urban traffic forecasting with contrastive pre-training](https://mlsanity.com/paper/2503.14980/) (2025)
- [Efficient Traffic State Prediction With Dynamic Joint Spatio-Temporal Relation Inference](https://mlsanity.com/paper/2504.08061/) (2025)
- [Lightweight and Interpretable Transformer via Mixed Graph Algorithm Unrolling for Traffic Forecast](https://mlsanity.com/paper/2505.13102/) (2025)
- [SPAR: Self-supervised Placement-Aware Representation Learning for Distributed Sensing](https://mlsanity.com/paper/2505.16936/) (2025)
- [Proxy Reconstruction Pre-training for Ramp Flow Prediction at Highway Interchanges](https://mlsanity.com/paper/2510.03381/) (2025)
- [Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Equality](https://mlsanity.com/paper/2511.03190/) (2025)
- [UniST-Pred: A Robust Unified Framework for Spatio-Temporal Traffic Forecasting in Transportation Networks Under Disruptions](https://mlsanity.com/paper/2602.14049/) (2026)
- [CoMemNet: A Continual Memory Network with Drift-Aware Sampling for Traffic Prediction](https://mlsanity.com/paper/2605.05738/) (2026)
- [Dimensional Balance Improves Large Scale Spatiotemporal Prediction Performance](https://mlsanity.com/paper/2605.18793/) (2026)
- [DSFNet: Learning Dual-Domain Spectral Operators for Multi-Modality Spatio-Temporal Forecasting in Urban Transportation Systems](https://mlsanity.com/paper/2606.07695/) (2026)
- [MP3: Multi-Period Pattern Pre-training for Spatio-Temporal Forecasting](https://mlsanity.com/paper/2606.13119/) (2026)
- [Global-Scale Self-Supervised Spatiotemporal Learning for NDVI Time-Series Reconstruction](https://mlsanity.com/paper/2608.02322/) (2026)
- [ButterMamba: Butterworth-Enhanced Spatial-Temporal Mamba for Efficient Traffic Flow Prediction](https://mlsanity.com/paper/2608.29658/) (2026)
- [TimeES: Probabilistic and Deterministic Time Series Forecasting via Evolutionary Spectra](https://mlsanity.com/paper/2609.32384/) (2026)

## Abstract (arXiv)

Spatiotemporal forecasting techniques are significant for various domains such as transportation, energy, and weather. Accurate prediction of spatiotemporal series remains challenging due to the complex spatiotemporal heterogeneity. In particular, current end-to-end models are limited by input length and thus often fall into spatiotemporal mirage, i.e., similar input time series followed by dissimilar future values and vice versa. To address these problems, we propose a novel self-supervised pre-training framework Spatial-Temporal-Decoupled Masked Pre-training (STD-MAE) that employs two decoupled masked autoencoders to reconstruct spatiotemporal series along the spatial and temporal dimensions. Rich-context representations learned through such reconstruction could be seamlessly integrated by downstream predictors with arbitrary architectures to augment their performances. A series of quantitative and qualitative evaluations on six widely used benchmarks (PEMS03, PEMS04, PEMS07, PEMS08, METR-LA, and PEMS-BAY) are conducted to validate the state-of-the-art performance of STD-MAE. Codes are available at https://github.com/Jimmy-7664/STD-MAE.