# Spectral Temporal Graph Neural Network for Trajectory Prediction

arXiv: https://arxiv.org/abs/2106.02930 · HTML: https://mlsanity.com/paper/2106.02930/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 5 Jun 2021.

## Builds on (most cited first, 30 of 36)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [GMAN: A Graph Multi-Attention Network for Traffic Prediction](https://mlsanity.com/paper/1911.08415/) (2019)
- [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Multivariate Time-series Anomaly Detection via Graph Attention Network](https://mlsanity.com/paper/2009.02040/) (2020)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs](https://mlsanity.com/paper/1805.05499/) (2018)
- [Conditional Generative Neural System for Probabilistic Trajectory Prediction](https://mlsanity.com/paper/1905.01631/) (2019)
- [Trajectory Forecasts in Unknown Environments Conditioned on Grid-Based Plans](https://mlsanity.com/paper/2001.00735/) (2020)
- [Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection](https://mlsanity.com/paper/1702.05552/) (2017)
- [Conditional Flow Variational Autoencoders for Structured Sequence Prediction](https://mlsanity.com/paper/1908.09008/) (2019)
- [Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships](https://mlsanity.com/paper/2011.04251/) (2020)
- [CAR-Net: Clairvoyant Attentive Recurrent Network](https://mlsanity.com/paper/1711.10061/) (2017)
- [Shared Cross-Modal Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2011.08436/) (2020)

## Built on by (most cited first, 30 of 36)

- [A Comprehensive Survey on Deep Graph Representation Learning](https://mlsanity.com/paper/2304.05055/) (2023)
- [Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction](https://mlsanity.com/paper/2305.14715/) (2023)
- [View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums](https://mlsanity.com/paper/2110.07288/) (2021)
- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2310.05370/) (2023)
- [MSN: Multi-Style Network for Trajectory Prediction](https://mlsanity.com/paper/2107.00932/) (2021)
- [Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networks](https://mlsanity.com/paper/2107.10234/) (2021)
- [Estimating Treatment Effects from Irregular Time Series Observations with Hidden Confounders](https://mlsanity.com/paper/2303.02320/) (2023)
- [TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks](https://mlsanity.com/paper/2602.13272/) (2026)
- [Another Vertical View: A Hierarchical Network for Heterogeneous Trajectory Prediction via Spectrums](https://mlsanity.com/paper/2304.05106/) (2023)
- [Scene Informer: Anchor-based Occlusion Inference and Trajectory Prediction in Partially Observable Environments](https://mlsanity.com/paper/2309.13893/) (2023)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Reliable Trajectory Prediction and Uncertainty Quantification with Conditioned Diffusion Models](https://mlsanity.com/paper/2405.14384/) (2024)
- [Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints](https://mlsanity.com/paper/2306.01075/) (2023)
- [Coupled Multiwavelet Neural Operator Learning for Coupled Partial Differential Equations](https://mlsanity.com/paper/2303.02304/) (2023)
- [Holistic Graph-based Motion Prediction](https://mlsanity.com/paper/2301.13545/) (2023)
- [A Multidimensional Graph Fourier Transformation Neural Network for Vehicle Trajectory Prediction](https://mlsanity.com/paper/2305.07416/) (2023)
- [SocialCircle+: Learning the Angle-based Conditioned Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2409.14984/) (2024)
- [DSLOB: A Synthetic Limit Order Book Dataset for Benchmarking Forecasting Algorithms under Distributional Shift](https://mlsanity.com/paper/2211.11513/) (2022)
- [Equivariant Map and Agent Geometry for Autonomous Driving Motion Prediction](https://mlsanity.com/paper/2310.13922/) (2023)
- [Certified Human Trajectory Prediction](https://mlsanity.com/paper/2403.13778/) (2024)
- [SocialFormer: Social Interaction Modeling with Edge-enhanced Heterogeneous Graph Transformers for Trajectory Prediction](https://mlsanity.com/paper/2405.03809/) (2024)
- [TrajEvo: Trajectory Prediction Heuristics Design via LLM-driven Evolution](https://mlsanity.com/paper/2508.05616/) (2025)
- [Towards trustworthy multi-modal motion prediction: Holistic evaluation and interpretability of outputs](https://mlsanity.com/paper/2210.16144/) (2022)
- [Prediction and Interpretation of Vehicle Trajectories in the Graph Spectral Domain](https://mlsanity.com/paper/2309.16702/) (2023)
- [Optimal Solving of Constrained Path-Planning Problems with Graph Convolutional Networks and Optimized Tree Search](https://mlsanity.com/paper/2108.01036/) (2021)
- [Conquering Ghosts: Relation Learning for Information Reliability Representation and End-to-End Robust Navigation](https://mlsanity.com/paper/2203.09952/) (2022)
- [Spectral GNN via Two-dimensional (2-D) Graph Convolution](https://mlsanity.com/paper/2404.04559/) (2024)
- [Who Walks With You Matters: Perceiving Social Interactions with Groups for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2412.02395/) (2024)
- [TrajLearn: Trajectory Prediction Learning using Deep Generative Models](https://mlsanity.com/paper/2501.00184/) (2024)

## Abstract (arXiv)

An effective understanding of the contextual environment and accurate motion forecasting of surrounding agents is crucial for the development of autonomous vehicles and social mobile robots. This task is challenging since the behavior of an autonomous agent is not only affected by its own intention, but also by the static environment and surrounding dynamically interacting agents. Previous works focused on utilizing the spatial and temporal information in time domain while not sufficiently taking advantage of the cues in frequency domain. To this end, we propose a Spectral Temporal Graph Neural Network (SpecTGNN), which can capture inter-agent correlations and temporal dependency simultaneously in frequency domain in addition to time domain. SpecTGNN operates on both an agent graph with dynamic state information and an environment graph with the features extracted from context images in two streams. The model integrates graph Fourier transform, spectral graph convolution and temporal gated convolution to encode history information and forecast future trajectories. Moreover, we incorporate a multi-head spatio-temporal attention mechanism to mitigate the effect of error propagation in a long time horizon. We demonstrate the performance of SpecTGNN on two public trajectory prediction benchmark datasets, which achieves state-of-the-art performance in terms of prediction accuracy.