# Multi-Modality Spatio-Temporal Forecasting via Self-Supervised Learning

arXiv: https://arxiv.org/abs/2405.03255 · HTML: https://mlsanity.com/paper/2405.03255/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting](https://mlsanity.com/paper/2007.02842/) (2020)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [GMAN: A Graph Multi-Attention Network for Traffic Prediction](https://mlsanity.com/paper/1911.08415/) (2019)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Dish-TS: A General Paradigm for Alleviating Distribution Shift in Time Series Forecasting](https://mlsanity.com/paper/2302.14829/) (2023)
- [Spatio-Temporal Meta-Graph Learning for Traffic Forecasting](https://mlsanity.com/paper/2211.14701/) (2022)
- [Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction](https://mlsanity.com/paper/2212.04475/) (2022)
- [DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction](https://mlsanity.com/paper/2108.09091/) (2021)
- [FC-GAGA: Fully Connected Gated Graph Architecture for Spatio-Temporal Traffic Forecasting](https://mlsanity.com/paper/2007.15531/) (2020)
- [DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting](https://mlsanity.com/paper/2203.07681/) (2022)
- [Automated Spatio-Temporal Graph Contrastive Learning](https://mlsanity.com/paper/2305.03920/) (2023)
- [MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data](https://mlsanity.com/paper/2110.14354/) (2021)
- [Space Meets Time: Local Spacetime Neural Network For Traffic Flow Forecasting](https://mlsanity.com/paper/2109.05225/) (2021)
- [Learning Gaussian Mixture Representations for Tensor Time Series Forecasting](https://mlsanity.com/paper/2306.00390/) (2023)

## Built on by (most cited first, 9 of 9)

- [Merlin: Multi-View Representation Learning for Robust Multivariate Time Series Forecasting with Unfixed Missing Rates](https://mlsanity.com/paper/2506.12459/) (2025)
- [Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review](https://mlsanity.com/paper/2506.01364/) (2025)
- [From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signals](https://mlsanity.com/paper/2405.16083/) (2024)
- [How Different from the Past? Spatio-Temporal Time Series Forecasting with Self-Supervised Deviation Learning](https://mlsanity.com/paper/2510.04908/) (2025)
- [Cross Space and Time: A Spatio-Temporal Unitized Model for Traffic Flow Forecasting](https://mlsanity.com/paper/2411.09251/) (2024)
- [Unveiling the Inflexibility of Adaptive Embedding in Traffic Forecasting](https://mlsanity.com/paper/2411.11448/) (2024)
- [DSFNet: Learning Dual-Domain Spectral Operators for Multi-Modality Spatio-Temporal Forecasting in Urban Transportation Systems](https://mlsanity.com/paper/2606.07695/) (2026)
- [MP3: Multi-Period Pattern Pre-training for Spatio-Temporal Forecasting](https://mlsanity.com/paper/2606.13119/) (2026)
- [Neural Architecture Search for Traffic Prediction: A Survey of Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2607.26467/) (2026)

## Abstract (arXiv)

Multi-modality spatio-temporal (MoST) data extends spatio-temporal (ST) data by incorporating multiple modalities, which is prevalent in monitoring systems, encompassing diverse traffic demands and air quality assessments. Despite significant strides in ST modeling in recent years, there remains a need to emphasize harnessing the potential of information from different modalities. Robust MoST forecasting is more challenging because it possesses (i) high-dimensional and complex internal structures and (ii) dynamic heterogeneity caused by temporal, spatial, and modality variations. In this study, we propose a novel MoST learning framework via Self-Supervised Learning, namely MoSSL, which aims to uncover latent patterns from temporal, spatial, and modality perspectives while quantifying dynamic heterogeneity. Experiment results on two real-world MoST datasets verify the superiority of our approach compared with the state-of-the-art baselines. Model implementation is available at https://github.com/beginner-sketch/MoSSL.