# Multi-modal anticipation of stochastic trajectories in a dynamic environment with Conditional Variational Autoencoders

arXiv: https://arxiv.org/abs/2103.03912 · HTML: https://mlsanity.com/paper/2103.03912/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [Hierarchical Multi-Scale Attention for Semantic Segmentation](https://mlsanity.com/paper/2005.10821/) (2020)
- [Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence](https://mlsanity.com/paper/2002.10542/) (2020)
- [Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving](https://mlsanity.com/paper/1808.05819/) (2018)
- [TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions](https://mlsanity.com/paper/1812.04767/) (2018)
- [MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction](https://mlsanity.com/paper/2006.03340/) (2020)
- [Fixing the train-test resolution discrepancy: FixEfficientNet](https://mlsanity.com/paper/2003.08237/) (2020)
- [Joint Interaction and Trajectory Prediction for Autonomous Driving using Graph Neural Networks](https://mlsanity.com/paper/1912.07882/) (2019)
- [INFER: INtermediate representations for FuturE pRediction](https://mlsanity.com/paper/1903.10641/) (2019)
- [DROGON: A Trajectory Prediction Model based on Intention-Conditioned Behavior Reasoning](https://mlsanity.com/paper/1908.00024/) (2019)
- [Exploiting latent representation of sparse semantic layers for improved short-term motion prediction with Capsule Networks](https://mlsanity.com/paper/2103.01644/) (2021)

## Built on by (most cited first, 1 of 1)

- [Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions](https://mlsanity.com/paper/2307.07527/) (2023)

## Abstract (arXiv)

Forecasting short-term motion of nearby vehicles presents an inherently challenging issue as the space of their possible future movements is not strictly limited to a set of single trajectories. Recently proposed techniques that demonstrate plausible results concentrate primarily on forecasting a fixed number of deterministic predictions, or on classifying over a wide variety of trajectories that were previously generated using e.g. dynamic model. This paper focuses on addressing the uncertainty associated with the discussed task by utilising the stochastic nature of generative models in order to produce a diverse set of plausible paths with regards to tracked vehicles. More specifically, we propose to account for the multi-modality of the problem with use of Conditional Variational Autoencoder (C-VAE) conditioned on an agent's past motion as well as a rasterised scene context encoded with Capsule Network (CapsNet). In addition, we demonstrate advantages of employing the Minimum over N (MoN) cost function which measures the distance between ground truth and N generated samples and tries to minimise the loss with respect to the closest sample, effectively leading to more diverse predictions. We examine our network on a publicly available dataset against recent state-of-the-art methods and show that our approach outperforms these techniques in numerous scenarios whilst significantly reducing the number of trainable parameters as well as allowing to sample an arbitrary amount of diverse trajectories.