# Exploiting latent representation of sparse semantic layers for improved short-term motion prediction with Capsule Networks

arXiv: https://arxiv.org/abs/2103.01644 · HTML: https://mlsanity.com/paper/2103.01644/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [Hierarchical Multi-Scale Attention for Semantic Segmentation](https://mlsanity.com/paper/2005.10821/) (2020)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction](https://mlsanity.com/paper/2006.03340/) (2020)
- [RODEO: Replay for Online Object Detection](https://mlsanity.com/paper/2008.06439/) (2020)
- [Fixing the train-test resolution discrepancy: FixEfficientNet](https://mlsanity.com/paper/2003.08237/) (2020)
- [INFER: INtermediate representations for FuturE pRediction](https://mlsanity.com/paper/1903.10641/) (2019)
- [DROGON: A Trajectory Prediction Model based on Intention-Conditioned Behavior Reasoning](https://mlsanity.com/paper/1908.00024/) (2019)

## Built on by (most cited first, 2 of 2)

- [Multi-modal anticipation of stochastic trajectories in a dynamic environment with Conditional Variational Autoencoders](https://mlsanity.com/paper/2103.03912/) (2021)
- [Towards the Characterization of Representations Learned via Capsule-based Network Architectures](https://mlsanity.com/paper/2305.05349/) (2023)

## Abstract (arXiv)

As urban environments manifest high levels of complexity it is of vital importance that safety systems embedded within autonomous vehicles (AVs) are able to accurately anticipate short-term future motion of nearby agents. This problem can be further understood as generating a sequence of coordinates describing the future motion of the tracked agent. Various proposed approaches demonstrate significant benefits of using a rasterised top-down image of the road, with a combination of Convolutional Neural Networks (CNNs), for extraction of relevant features that define the road structure (eg. driveable areas, lanes, walkways). In contrast, this paper explores use of Capsule Networks (CapsNets) in the context of learning a hierarchical representation of sparse semantic layers corresponding to small regions of the High-Definition (HD) map. Each region of the map is dismantled into separate geometrical layers that are extracted with respect to the agent's current position. By using an architecture based on CapsNets the model is able to retain hierarchical relationships between detected features within images whilst also preventing loss of spatial data often caused by the pooling operation. We train and evaluate our model on publicly available dataset nuTonomy scenes and compare it to recently published methods. We show that our model achieves significant improvement over recently published works on deterministic prediction, whilst drastically reducing the overall size of the network.