# Learning dynamic and hierarchical traffic spatiotemporal features with Transformer

arXiv: https://arxiv.org/abs/2104.05163 · HTML: https://mlsanity.com/paper/2104.05163/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [GMAN: A Graph Multi-Attention Network for Traffic Prediction](https://mlsanity.com/paper/1911.08415/) (2019)
- [GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs](https://mlsanity.com/paper/1803.07294/) (2018)
- [Learning Traffic as Images: A Deep Convolutional Neural Network for Large-Scale Transportation Network Speed Prediction](https://mlsanity.com/paper/1701.04245/) (2017)
- [Spatiotemporal Recurrent Convolutional Networks for Traffic Prediction in Transportation Networks](https://mlsanity.com/paper/1705.02699/) (2017)
- [ST-UNet: A Spatio-Temporal U-Network for Graph-structured Time Series Modeling](https://mlsanity.com/paper/1903.05631/) (2019)

## Built on by (most cited first, 26 of 26)

- [PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction](https://mlsanity.com/paper/2301.07945/) (2023)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [Context-aware multi-head self-attentional neural network model for next location prediction](https://mlsanity.com/paper/2212.01953/) (2022)
- [FollowNet: A Comprehensive Benchmark for Car-Following Behavior Modeling](https://mlsanity.com/paper/2306.05381/) (2023)
- [PromptST: Prompt-Enhanced Spatio-Temporal Multi-Attribute Prediction](https://mlsanity.com/paper/2309.09500/) (2023)
- [Rethinking Sensors Modeling: Hierarchical Information Enhanced Traffic Forecasting](https://mlsanity.com/paper/2309.11284/) (2023)
- [Adaptive Hierarchical SpatioTemporal Network for Traffic Forecasting](https://mlsanity.com/paper/2306.09386/) (2023)
- [Mlinear: Rethink the Linear Model for Time-series Forecasting](https://mlsanity.com/paper/2305.04800/) (2023)
- [Unifying Lane-Level Traffic Prediction from a Graph Structural Perspective: Benchmark and Baseline](https://mlsanity.com/paper/2403.14941/) (2024)
- [RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting](https://mlsanity.com/paper/2410.24023/) (2024)
- [Multi-Source Urban Traffic Flow Forecasting with Drone and Loop Detector Data](https://mlsanity.com/paper/2501.03492/) (2025)
- [Network Level Spatial Temporal Traffic State Forecasting with Hierarchical-Attention-LSTM (HierAttnLSTM)](https://mlsanity.com/paper/2201.05760/) (2022)
- [Traffic-Twitter Transformer: A Nature Language Processing-joined Framework For Network-wide Traffic Forecasting](https://mlsanity.com/paper/2206.11078/) (2022)
- [Unified Data Management and Comprehensive Performance Evaluation for Urban Spatial-Temporal Prediction [Experiment, Analysis & Benchmark]](https://mlsanity.com/paper/2308.12899/) (2023)
- [A novel hybrid time-varying graph neural network for traffic flow forecasting](https://mlsanity.com/paper/2401.10155/) (2024)
- [An Efficient Two-stage Gradient Boosting Framework for Short-term Traffic State Estimation](https://mlsanity.com/paper/2302.10400/) (2023)
- [Counterfactual Graph Transformer for Traffic Flow Prediction](https://mlsanity.com/paper/2308.00391/) (2023)
- [Lightweight Spatio-Temporal Attention Network with Graph Embedding and Rotational Position Encoding for Traffic Forecasting](https://mlsanity.com/paper/2505.12136/) (2025)
- [ADFormer: Aggregation Differential Transformer for Passenger Demand Forecasting](https://mlsanity.com/paper/2506.02576/) (2025)
- [ST-LINK: Spatially-Aware Large Language Models for Spatio-Temporal Forecasting](https://mlsanity.com/paper/2509.13753/) (2025)
- [ST-former for short-term passenger flow prediction during COVID-19 in urban rail transit system](https://mlsanity.com/paper/2210.09043/) (2022)
- [STTM: A New Approach Based Spatial-Temporal Transformer And Memory Network For Real-time Pressure Signal In On-demand Food Delivery](https://mlsanity.com/paper/2410.00057/) (2024)
- [Toward Dependency Dynamics in Multi-Agent Reinforcement Learning for Traffic Signal Control](https://mlsanity.com/paper/2502.16608/) (2025)
- [Rethinking Traffic Flow Forecasting: From Transition to Generatation](https://mlsanity.com/paper/2504.14248/) (2025)
- [The Geography of Transportation Cybersecurity: Visitor Flows, Industry Clusters, and Spatial Dynamics](https://mlsanity.com/paper/2505.08822/) (2025)
- [Long-Horizon Traffic Forecasting via Incident-Aware Conformal Spatio-Temporal Transformers](https://mlsanity.com/paper/2603.16857/) (2026)

## Abstract (arXiv)

Traffic forecasting is an indispensable part of Intelligent transportation systems (ITS), and long-term network-wide accurate traffic speed forecasting is one of the most challenging tasks. Recently, deep learning methods have become popular in this domain. As traffic data are physically associated with road networks, most proposed models treat it as a spatiotemporal graph modeling problem and use Graph Convolution Network (GCN) based methods. These GCN-based models highly depend on a predefined and fixed adjacent matrix to reflect the spatial dependency. However, the predefined fixed adjacent matrix is limited in reflecting the actual dependence of traffic flow. This paper proposes a novel model, Traffic Transformer, for spatial-temporal graph modeling and long-term traffic forecasting to overcome these limitations. Transformer is the most popular framework in Natural Language Processing (NLP). And by adapting it to the spatiotemporal problem, Traffic Transformer hierarchically extracts spatiotemporal features through data dynamically by multi-head attention and masked multi-head attention mechanism, and fuse these features for traffic forecasting. Furthermore, analyzing the attention weight matrixes can find the influential part of road networks, allowing us to learn the traffic networks better. Experimental results on the public traffic network datasets and real-world traffic network datasets generated by ourselves demonstrate our proposed model achieves better performance than the state-of-the-art ones.