# Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking

arXiv: https://arxiv.org/abs/2102.09117 · HTML: https://mlsanity.com/paper/2102.09117/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 60 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents](https://mlsanity.com/paper/1811.02146/) (2018)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)
- [Conditional Generative Neural System for Probabilistic Trajectory Prediction](https://mlsanity.com/paper/1905.01631/) (2019)
- [Relational Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01830/) (2018)
- [Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding](https://mlsanity.com/paper/2003.03212/) (2020)
- [How would surround vehicles move? A Unified Framework for Maneuver Classification and Motion Prediction](https://mlsanity.com/paper/1801.06523/) (2018)
- [Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection](https://mlsanity.com/paper/1702.05552/) (2017)

## Built on by (most cited first, 30 of 35)

- [Multi-Person Extreme Motion Prediction](https://mlsanity.com/paper/2105.08825/) (2021)
- [RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting](https://mlsanity.com/paper/2108.01316/) (2021)
- [CMP: Cooperative Motion Prediction with Multi-Agent Communication](https://mlsanity.com/paper/2403.17916/) (2024)
- [Grouptron: Dynamic Multi-Scale Graph Convolutional Networks for Group-Aware Dense Crowd Trajectory Forecasting](https://mlsanity.com/paper/2109.14128/) (2021)
- [MTP-GO: Graph-Based Probabilistic Multi-Agent Trajectory Prediction with Neural ODEs](https://mlsanity.com/paper/2302.00735/) (2023)
- [Pedestrian Trajectory Prediction in Pedestrian-Vehicle Mixed Environments: A Systematic Review](https://mlsanity.com/paper/2308.06419/) (2023)
- [Interaction Modeling with Multiplex Attention](https://mlsanity.com/paper/2208.10660/) (2022)
- [Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration](https://mlsanity.com/paper/2309.09408/) (2023)
- [Scene Informer: Anchor-based Occlusion Inference and Trajectory Prediction in Partially Observable Environments](https://mlsanity.com/paper/2309.13893/) (2023)
- [Diffusion-Based Environment-Aware Trajectory Prediction](https://mlsanity.com/paper/2403.11643/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Important Object Identification with Semi-Supervised Learning for Autonomous Driving](https://mlsanity.com/paper/2203.02634/) (2022)
- [Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2207.05195/) (2022)
- [Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation](https://mlsanity.com/paper/2311.16091/) (2023)
- [MATRIX: Multi-Agent Trajectory Generation with Diverse Contexts](https://mlsanity.com/paper/2403.06041/) (2024)
- [Bridging the Gap Between Multi-Step and One-Shot Trajectory Prediction via Self-Supervision](https://mlsanity.com/paper/2306.03367/) (2023)
- [EqDrive: Efficient Equivariant Motion Forecasting with Multi-Modality for Autonomous Driving](https://mlsanity.com/paper/2310.17540/) (2023)
- [Disentangled Neural Relational Inference for Interpretable Motion Prediction](https://mlsanity.com/paper/2401.03599/) (2024)
- [Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecasting](https://mlsanity.com/paper/2407.09475/) (2024)
- [Relation-based Motion Prediction using Traffic Scene Graphs](https://mlsanity.com/paper/2212.02503/) (2022)
- [Evaluation of Differentially Constrained Motion Models for Graph-Based Trajectory Prediction](https://mlsanity.com/paper/2304.05116/) (2023)
- [Equivariant Map and Agent Geometry for Autonomous Driving Motion Prediction](https://mlsanity.com/paper/2310.13922/) (2023)
- [Towards Generalizable Safety in Crowd Navigation via Conformal Uncertainty Handling](https://mlsanity.com/paper/2508.05634/) (2025)
- [TrajEvo: Trajectory Prediction Heuristics Design via LLM-driven Evolution](https://mlsanity.com/paper/2508.05616/) (2025)
- [TrajPRed: Trajectory Prediction with Region-based Relation Learning](https://mlsanity.com/paper/2404.06971/) (2024)
- [Heterogeneous Interaction Modeling With Reduced Accumulated Error for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2410.21342/) (2024)
- [Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2509.10570/) (2025)
- [Polar Collision Grids: Effective Interaction Modelling for Pedestrian Trajectory Prediction in Shared Space Using Collision Checks](https://mlsanity.com/paper/2308.06654/) (2023)
- [IR-STP: Enhancing Autonomous Driving with Interaction Reasoning in Spatio-Temporal Planning](https://mlsanity.com/paper/2311.02850/) (2023)
- [Enhanced Prediction of Multi-Agent Trajectories via Control Inference and State-Space Dynamics](https://mlsanity.com/paper/2408.12609/) (2024)

## Abstract (arXiv)

An effective understanding of the environment and accurate trajectory prediction of surrounding dynamic obstacles are indispensable for intelligent mobile systems (e.g. autonomous vehicles and social robots) to achieve safe and high-quality planning when they navigate in highly interactive and crowded scenarios. Due to the existence of frequent interactions and uncertainty in the scene evolution, it is desired for the prediction system to enable relational reasoning on different entities and provide a distribution of future trajectories for each agent. In this paper, we propose a generic generative neural system (called STG-DAT) for multi-agent trajectory prediction involving heterogeneous agents. The system takes a step forward to explicit interaction modeling by incorporating relational inductive biases with a dynamic graph representation and leverages both trajectory and scene context information. We also employ an efficient kinematic constraint layer applied to vehicle trajectory prediction. The constraint not only ensures physical feasibility but also enhances model performance. Moreover, the proposed prediction model can be easily adopted by multi-target tracking frameworks. The tracking accuracy proves to be improved by empirical results. The proposed system is evaluated on three public benchmark datasets for trajectory prediction, where the agents cover pedestrians, cyclists and on-road vehicles. The experimental results demonstrate that our model achieves better performance than various baseline approaches in terms of prediction and tracking accuracy.