# S2TNet: Spatio-Temporal Transformer Networks for Trajectory Prediction in Autonomous Driving

arXiv: https://arxiv.org/abs/2206.10902 · HTML: https://mlsanity.com/paper/2206.10902/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 22 Jun 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents](https://mlsanity.com/paper/1811.02146/) (2018)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)
- [TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions](https://mlsanity.com/paper/1812.04767/) (2018)
- [TPNet: Trajectory Proposal Network for Motion Prediction](https://mlsanity.com/paper/2004.12255/) (2020)
- [GRIP++: Enhanced Graph-based Interaction-aware Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/1907.07792/) (2019)
- [StarNet: Pedestrian Trajectory Prediction using Deep Neural Network in Star Topology](https://mlsanity.com/paper/1906.01797/) (2019)

## Built on by (most cited first, 22 of 22)

- [UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction](https://mlsanity.com/paper/2402.11838/) (2024)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions](https://mlsanity.com/paper/2307.07527/) (2023)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Attention-aware Social Graph Transformer Networks for Stochastic Trajectory Prediction](https://mlsanity.com/paper/2312.15881/) (2023)
- [UniFlow: A Foundation Model for Unified Urban Spatio-Temporal Flow Prediction](https://mlsanity.com/paper/2411.12972/) (2024)
- [Pose2Trajectory: Using Transformers on Body Pose to Predict Tennis Player's Trajectory](https://mlsanity.com/paper/2411.04501/) (2024)
- [Beyond Patterns: Harnessing Causal Logic for Autonomous Driving Trajectory Prediction](https://mlsanity.com/paper/2505.06856/) (2025)
- [Beyond Frame-wise Tracking: A Trajectory-based Paradigm for Efficient Point Cloud Tracking](https://mlsanity.com/paper/2509.11453/) (2025)
- [FollowGen: A Scaled Noise Conditional Diffusion Model for Car-Following Trajectory Prediction](https://mlsanity.com/paper/2411.16747/) (2024)
- [Crossfusor: A Cross-Attention Transformer Enhanced Conditional Diffusion Model for Car-Following Trajectory Prediction](https://mlsanity.com/paper/2406.11941/) (2024)
- [ASTRA: A Scene-aware TRAnsformer-based model for trajectory prediction](https://mlsanity.com/paper/2501.09878/) (2025)
- [SafeCast: Risk-Responsive Motion Forecasting for Autonomous Vehicles](https://mlsanity.com/paper/2503.22541/) (2025)
- [Multi-modal Spatio-Temporal Transformer for High-resolution Land Subsidence Prediction](https://mlsanity.com/paper/2509.25393/) (2025)
- [SARN: Structurally-Aware Recurrent Network for Spatio-Temporal Disaggregation](https://mlsanity.com/paper/2306.07292/) (2023)
- [STF: Spatial Temporal Fusion for Trajectory Prediction](https://mlsanity.com/paper/2311.18149/) (2023)
- [Short-term Inland Vessel Trajectory Prediction with Encoder-Decoder Models](https://mlsanity.com/paper/2406.02770/) (2024)
- [C2INet: Realizing Incremental Trajectory Prediction with Prior-Aware Continual Causal Intervention](https://mlsanity.com/paper/2411.12313/) (2024)
- [Collaborative Deterministic-Probabilistic Forecasting for Diverse Spatiotemporal Systems](https://mlsanity.com/paper/2502.11013/) (2025)
- [Towards Human-Like Trajectory Prediction for Autonomous Driving: A Behavior-Centric Approach](https://mlsanity.com/paper/2505.21565/) (2025)
- [Forecasting the Past: Gradient-Based Distribution Shift Detection in Trajectory Prediction](https://mlsanity.com/paper/2604.12425/) (2026)
- [Long-Term Prediction of Local and Global Human Motion with Occlusion Recovery](https://mlsanity.com/paper/2606.27900/) (2026)

## Abstract (arXiv)

To safely and rationally participate in dense and heterogeneous traffic, autonomous vehicles require to sufficiently analyze the motion patterns of surrounding traffic-agents and accurately predict their future trajectories. This is challenging because the trajectories of traffic-agents are not only influenced by the traffic-agents themselves but also by spatial interaction with each other. Previous methods usually rely on the sequential step-by-step processing of Long Short-Term Memory networks (LSTMs) and merely extract the interactions between spatial neighbors for single type traffic-agents. We propose the Spatio-Temporal Transformer Networks (S2TNet), which models the spatio-temporal interactions by spatio-temporal Transformer and deals with the temporel sequences by temporal Transformer. We input additional category, shape and heading information into our networks to handle the heterogeneity of traffic-agents. The proposed methods outperforms state-of-the-art methods on ApolloScape Trajectory dataset by more than 7% on both the weighted sum of Average and Final Displacement Error. Our code is available at https://github.com/chenghuang66/s2tnet.