# Multimodal Manoeuvre and Trajectory Prediction for Automated Driving on Highways Using Transformer Networks

arXiv: https://arxiv.org/abs/2303.16109 · HTML: https://mlsanity.com/paper/2303.16109/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMs](https://mlsanity.com/paper/1805.05499/) (2018)
- [Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving](https://mlsanity.com/paper/2109.06446/) (2021)
- [LaPred: Lane-Aware Prediction of Multi-Modal Future Trajectories of Dynamic Agents](https://mlsanity.com/paper/2104.00249/) (2021)
- [ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning](https://mlsanity.com/paper/2206.13387/) (2022)
- [Map-Adaptive Goal-Based Trajectory Prediction](https://mlsanity.com/paper/2009.04450/) (2020)
- [Trajectory Prediction in Autonomous Driving with a Lane Heading Auxiliary Loss](https://mlsanity.com/paper/2011.06679/) (2020)
- [Naturalistic Driver Intention and Path Prediction using Recurrent Neural Networks](https://mlsanity.com/paper/1807.09995/) (2018)
- [A Survey on Deep-Learning Approaches for Vehicle Trajectory Prediction in Autonomous Driving](https://mlsanity.com/paper/2110.10436/) (2021)
- [Maneuver-based Trajectory Prediction for Self-driving Cars Using Spatio-temporal Convolutional Networks](https://mlsanity.com/paper/2109.07365/) (2021)
- [Early Lane Change Prediction for Automated Driving Systems Using Multi-Task Attention-based Convolutional Neural Networks](https://mlsanity.com/paper/2109.10742/) (2021)
- [Teaching Vehicles to Anticipate: A Systematic Study on Probabilistic Behavior Prediction Using Large Data Sets](https://mlsanity.com/paper/1910.07772/) (2019)

## Built on by (most cited first, 7 of 7)

- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [A Novel Deep Neural Network for Trajectory Prediction in Automated Vehicles Using Velocity Vector Field](https://mlsanity.com/paper/2309.10948/) (2023)
- [SwapTransformer: highway overtaking tactical planner model via imitation learning on OSHA dataset](https://mlsanity.com/paper/2401.01425/) (2024)
- [Toward Unified Practices in Trajectory Prediction Research on Bird's-Eye-View Datasets](https://mlsanity.com/paper/2405.00604/) (2024)
- [LitSim: A Conflict-aware Policy for Long-term Interactive Traffic Simulation](https://mlsanity.com/paper/2403.04299/) (2024)
- [Interpretable Interaction Modeling for Trajectory Prediction via Agent Selection and Physical Coefficient](https://mlsanity.com/paper/2405.13152/) (2024)
- [Towards A General-Purpose Motion Planning for Autonomous Vehicles Using Fluid Dynamics](https://mlsanity.com/paper/2406.05708/) (2024)

## Abstract (arXiv)

Predicting the behaviour (i.e., manoeuvre/trajectory) of other road users, including vehicles, is critical for the safe and efficient operation of autonomous vehicles (AVs), a.k.a., automated driving systems (ADSs). Due to the uncertain future behaviour of vehicles, multiple future behaviour modes are often plausible for a vehicle in a given driving scene. Therefore, multimodal prediction can provide richer information than single-mode prediction, enabling AVs to perform a better risk assessment. To this end, we propose a novel multimodal prediction framework that can predict multiple plausible behaviour modes and their likelihoods. The proposed framework includes a bespoke problem formulation for manoeuvre prediction, a novel transformer-based prediction model, and a tailored training method for multimodal manoeuvre and trajectory prediction. The performance of the framework is evaluated using three public highway driving datasets, namely NGSIM, highD, and exiD. The results show that our framework outperforms the state-of-the-art multimodal methods in terms of prediction error and is capable of predicting plausible manoeuvre and trajectory modes.