# Under the Hood of Transformer Networks for Trajectory Forecasting

arXiv: https://arxiv.org/abs/2203.11878 · HTML: https://mlsanity.com/paper/2203.11878/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [DLow: Diversifying Latent Flows for Diverse Human Motion Prediction](https://mlsanity.com/paper/2003.08386/) (2020)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction](https://mlsanity.com/paper/1903.07933/) (2019)
- [Diverse Trajectory Forecasting with Determinantal Point Processes](https://mlsanity.com/paper/1907.04967/) (2019)
- [Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies](https://mlsanity.com/paper/1701.01909/) (2017)
- [The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction](https://mlsanity.com/paper/1912.06445/) (2019)
- [Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data](https://mlsanity.com/paper/2001.03093/) (2020)
- [End-to-end Contextual Perception and Prediction with Interaction Transformer](https://mlsanity.com/paper/2008.05927/) (2020)
- [Divide-and-Conquer for Lane-Aware Diverse Trajectory Prediction](https://mlsanity.com/paper/2104.08277/) (2021)
- [Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision](https://mlsanity.com/paper/1911.01138/) (2019)
- [OpenTraj: Assessing Prediction Complexity in Human Trajectories Datasets](https://mlsanity.com/paper/2010.00890/) (2020)
- [Pedestrian Trajectory Prediction with Convolutional Neural Networks](https://mlsanity.com/paper/2010.05796/) (2020)
- [CSCNet: Contextual Semantic Consistency Network for Trajectory Prediction in Crowded Spaces](https://mlsanity.com/paper/2202.08506/) (2022)
- [MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses](https://mlsanity.com/paper/1805.00652/) (2018)

## Built on by (most cited first, 8 of 8)

- [Certified Human Trajectory Prediction](https://mlsanity.com/paper/2403.13778/) (2024)
- [Distilling Knowledge for Short-to-Long Term Trajectory Prediction](https://mlsanity.com/paper/2305.08553/) (2023)
- [Toward Smart Doors: A Position Paper](https://mlsanity.com/paper/2209.11770/) (2022)
- [A Data-driven Crowd Simulation Framework Integrating Physics-informed Machine Learning with Navigation Potential Fields](https://mlsanity.com/paper/2410.16132/) (2024)
- [ECAM: A Contrastive Learning Approach to Avoid Environmental Collision in Trajectory Forecasting](https://mlsanity.com/paper/2506.09626/) (2025)
- [TrajGATFormer: A Graph-Based Transformer Approach for Worker and Obstacle Trajectory Prediction in Off-site Construction Environments](https://mlsanity.com/paper/2510.22205/) (2025)
- [Pedestrian Motion Prediction Using Transformer-based Behavior Clustering and Data-Driven Reachability Analysis](https://mlsanity.com/paper/2408.15250/) (2024)
- [Patient-specific vs Multi-Patient Vision Transformer for Markerless Tumor Motion Forecasting](https://mlsanity.com/paper/2507.07811/) (2025)

## Abstract (arXiv)

Transformer Networks have established themselves as the de-facto state-of-the-art for trajectory forecasting but there is currently no systematic study on their capability to model the motion patterns of people, without interactions with other individuals nor the social context. This paper proposes the first in-depth study of Transformer Networks (TF) and Bidirectional Transformers (BERT) for the forecasting of the individual motion of people, without bells and whistles. We conduct an exhaustive evaluation of input/output representations, problem formulations and sequence modeling, including a novel analysis of their capability to predict multi-modal futures. Out of comparative evaluation on the ETH+UCY benchmark, both TF and BERT are top performers in predicting individual motions, definitely overcoming RNNs and LSTMs. Furthermore, they remain within a narrow margin wrt more complex techniques, which include both social interactions and scene contexts. Source code will be released for all conducted experiments.