# TSGN: Temporal Scene Graph Neural Networks with Projected Vectorized Representation for Multi-Agent Motion Prediction

arXiv: https://arxiv.org/abs/2305.08190 · HTML: https://mlsanity.com/paper/2305.08190/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 14 May 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](https://mlsanity.com/paper/2106.15004/) (2021)
- [HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding](https://mlsanity.com/paper/2205.09753/) (2022)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [TPCN: Temporal Point Cloud Networks for Motion Forecasting](https://mlsanity.com/paper/2103.03067/) (2021)
- [Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving](https://mlsanity.com/paper/1808.05819/) (2018)
- [Learning an Uncertainty-Aware Object Detector for Autonomous Driving](https://mlsanity.com/paper/1910.11375/) (2019)
- [Bootstrap Motion Forecasting With Self-Consistent Constraints](https://mlsanity.com/paper/2204.05859/) (2022)

## Built on by (most cited first, 2 of 2)

- [MSTF: Multiscale Transformer for Incomplete Trajectory Prediction](https://mlsanity.com/paper/2407.05671/) (2024)
- [SSP-DMGTimeNet: Physics-Constrained Learning for Spatiotemporal Trajectory Prediction of Vehicle Platoons](https://mlsanity.com/paper/2609.06961/) (2026)

## Abstract (arXiv)

Predicting future motions of nearby agents is essential for an autonomous vehicle to take safe and effective actions. In this paper, we propose TSGN, a framework using Temporal Scene Graph Neural Networks with projected vectorized representations for multi-agent trajectory prediction. Projected vectorized representation models the traffic scene as a graph which is constructed by a set of vectors. These vectors represent agents, road network, and their spatial relative relationships. All relative features under this representation are both translationand rotation-invariant. Based on this representation, TSGN captures the spatial-temporal features across agents, road network, interactions among them, and temporal dependencies of temporal traffic scenes. TSGN can predict multimodal future trajectories for all agents simultaneously, plausibly, and accurately. Meanwhile, we propose a Hierarchical Lane Transformer for capturing interactions between agents and road network, which filters the surrounding road network and only keeps the most probable lane segments which could have an impact on the future behavior of the target agent. Without sacrificing the prediction performance, this greatly reduces the computational burden. Experiments show TSGN achieves state-of-the-art performance on the Argoverse motion forecasting benchmar.