# Long-Range Transformers for Dynamic Spatiotemporal Forecasting

arXiv: https://arxiv.org/abs/2109.12218 · HTML: https://mlsanity.com/paper/2109.12218/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 108 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)

## Built on by (most cited first, 30 of 46)

- [Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators](https://mlsanity.com/paper/2401.17548/) (2024)
- [LightCTS: A Lightweight Framework for Correlated Time Series Forecasting](https://mlsanity.com/paper/2302.11974/) (2023)
- [Dynamic Graph Representation Learning with Neural Networks: A Survey](https://mlsanity.com/paper/2304.05729/) (2023)
- [SageFormer: Series-Aware Framework for Long-term Multivariate Time Series Forecasting](https://mlsanity.com/paper/2307.01616/) (2023)
- [Semi-Parametric Inducing Point Networks and Neural Processes](https://mlsanity.com/paper/2205.11718/) (2022)
- [Diagnostic Spatio-temporal Transformer with Faithful Encoding](https://mlsanity.com/paper/2305.17149/) (2023)
- [Attention for Causal Relationship Discovery from Biological Neural Dynamics](https://mlsanity.com/paper/2311.06928/) (2023)
- [Generalist Multimodal AI: A Review of Architectures, Challenges and Opportunities](https://mlsanity.com/paper/2406.05496/) (2024)
- [Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance](https://mlsanity.com/paper/2412.00979/) (2024)
- [A Novel Hybrid Approach Using an Attention-Based Transformer + GRU Model for Predicting Cryptocurrency Prices](https://mlsanity.com/paper/2504.17079/) (2025)
- [An End-to-End Time Series Model for Simultaneous Imputation and Forecast](https://mlsanity.com/paper/2306.00778/) (2023)
- [Can Attention Be Used to Explain EHR-Based Mortality Prediction Tasks: A Case Study on Hemorrhagic Stroke](https://mlsanity.com/paper/2308.05110/) (2023)
- [Graph Neural Networks for Electric and Hydraulic Data Fusion to Enhance Short-term Forecasting of Pumped-storage Hydroelectricity](https://mlsanity.com/paper/2404.03368/) (2024)
- [miMamba: EEG-based Emotion Recognition with Multi-scale Inverted Mamba Models](https://mlsanity.com/paper/2409.07589/) (2024)
- [D2Vformer: A Flexible Time Series Prediction Model Based on Time Position Embedding](https://mlsanity.com/paper/2409.11024/) (2024)
- [T-Graphormer: Using Transformers for Spatiotemporal Forecasting](https://mlsanity.com/paper/2501.13274/) (2025)
- [Position: The Inevitable End of One-Architecture-Fits-All-Domains in Time Series Forecasting](https://mlsanity.com/paper/2602.01736/) (2026)
- [UnetTSF: A Better Performance Linear Complexity Time Series Prediction Model](https://mlsanity.com/paper/2401.03001/) (2024)
- [Scalable Numerical Embeddings for Multivariate Time Series: Enhancing Healthcare Data Representation Learning](https://mlsanity.com/paper/2405.16557/) (2024)
- [Multi-View Neural Differential Equations for Continuous-Time Stream Data in Long-Term Traffic Forecasting](https://mlsanity.com/paper/2408.06445/) (2024)
- [FATE: Focal-modulated Attention Encoder for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2408.11336/) (2024)
- [Integrating the Expected Future in Load Forecasts with Contextually Enhanced Transformer Models](https://mlsanity.com/paper/2409.05884/) (2024)
- [On the Regularization of Learnable Embeddings for Time Series Forecasting](https://mlsanity.com/paper/2410.14630/) (2024)
- [TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting](https://mlsanity.com/paper/2410.20439/) (2024)
- [Industrial Machines Health Prognosis using a Transformer-based Framework](https://mlsanity.com/paper/2411.14443/) (2024)
- [Dynamic Modes as Time Representation for Spatiotemporal Forecasting](https://mlsanity.com/paper/2506.01212/) (2025)
- [LSDM: LLM-Enhanced Spatio-temporal Diffusion Model for Service-Level Mobile Traffic Prediction](https://mlsanity.com/paper/2507.17795/) (2025)
- [Towards Unified and Data-Efficient Prognostics and Health Management with Tabular Foundation Models](https://mlsanity.com/paper/2606.05481/) (2026)
- [A Transformer Framework for Data Fusion and Multi-Task Learning in Smart Cities](https://mlsanity.com/paper/2211.10506/) (2022)
- [A Concurrent CNN-RNN Approach for Multi-Step Wind Power Forecasting](https://mlsanity.com/paper/2301.00819/) (2023)

## Abstract (arXiv)

Multivariate time series forecasting focuses on predicting future values based on historical context. State-of-the-art sequence-to-sequence models rely on neural attention between timesteps, which allows for temporal learning but fails to consider distinct spatial relationships between variables. In contrast, methods based on graph neural networks explicitly model variable relationships. However, these methods often rely on predefined graphs that cannot change over time and perform separate spatial and temporal updates without establishing direct connections between each variable at every timestep. Our work addresses these problems by translating multivariate forecasting into a "spatiotemporal sequence" formulation where each Transformer input token represents the value of a single variable at a given time. Long-Range Transformers can then learn interactions between space, time, and value information jointly along this extended sequence. Our method, which we call Spacetimeformer, achieves competitive results on benchmarks from traffic forecasting to electricity demand and weather prediction while learning spatiotemporal relationships purely from data.