# Connected Autonomous Vehicle Motion Planning with Video Predictions from Smart, Self-Supervised Infrastructure

arXiv: https://arxiv.org/abs/2309.07504 · HTML: https://mlsanity.com/paper/2309.07504/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Cross-view Semantic Segmentation for Sensing Surroundings](https://mlsanity.com/paper/1906.03560/) (2019)
- [An Uncertain Future: Forecasting from Static Images using Variational Autoencoders](https://mlsanity.com/paper/1606.07873/) (2016)
- [Learning a Decision Module by Imitating Driver's Control Behaviors](https://mlsanity.com/paper/1912.00191/) (2019)
- [A Survey of Federated Learning for Connected and Automated Vehicles](https://mlsanity.com/paper/2303.10677/) (2023)
- [DiNNO: Distributed Neural Network Optimization for Multi-Robot Collaborative Learning](https://mlsanity.com/paper/2109.08665/) (2021)
- [Control of Connected and Automated Vehicles: State of the Art and Future Challenges](https://mlsanity.com/paper/1804.03757/) (2018)
- [Infrastructure Enabled Autonomy: A Distributed Intelligence Architecture for Autonomous Vehicles](https://mlsanity.com/paper/1802.04112/) (2018)
- [Self-Supervised Traffic Advisors: Distributed, Multi-view Traffic Prediction for Smart Cities](https://mlsanity.com/paper/2204.06171/) (2022)

## Abstract (arXiv)

Connected autonomous vehicles (CAVs) promise to enhance safety, efficiency, and sustainability in urban transportation. However, this is contingent upon a CAV correctly predicting the motion of surrounding agents and planning its own motion safely. Doing so is challenging in complex urban environments due to frequent occlusions and interactions among many agents. One solution is to leverage smart infrastructure to augment a CAV's situational awareness; the present work leverages a recently proposed "Self-Supervised Traffic Advisor" (SSTA) framework of smart sensors that teach themselves to generate and broadcast useful video predictions of road users. In this work, SSTA predictions are modified to predict future occupancy instead of raw video, which reduces the data footprint of broadcast predictions. The resulting predictions are used within a planning framework, demonstrating that this design can effectively aid CAV motion planning. A variety of numerical experiments study the key factors that make SSTA outputs useful for practical CAV planning in crowded urban environments.