# BEVSeg2TP: Surround View Camera Bird's-Eye-View Based Joint Vehicle Segmentation and Ego Vehicle Trajectory Prediction

arXiv: https://arxiv.org/abs/2312.13081 · HTML: https://mlsanity.com/paper/2312.13081/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [A survey of loss functions for semantic segmentation](https://mlsanity.com/paper/2006.14822/) (2020)
- [Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving](https://mlsanity.com/paper/1903.11444/) (2019)
- [Spherical Transformer for LiDAR-based 3D Recognition](https://mlsanity.com/paper/2303.12766/) (2023)
- [ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape](https://mlsanity.com/paper/1812.02781/) (2018)
- [3D-LaneNet: End-to-End 3D Multiple Lane Detection](https://mlsanity.com/paper/1811.10203/) (2018)
- [Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving](https://mlsanity.com/paper/1808.05819/) (2018)
- [Surround-view Fisheye Camera Perception for Automated Driving: Overview, Survey and Challenges](https://mlsanity.com/paper/2205.13281/) (2022)
- [Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving](https://mlsanity.com/paper/2105.12713/) (2021)
- [Near-field Perception for Low-Speed Vehicle Automation using Surround-view Fisheye Cameras](https://mlsanity.com/paper/2103.17001/) (2021)
- [Monocular 3D Vehicle Detection Using Uncalibrated Traffic Cameras through Homography](https://mlsanity.com/paper/2103.15293/) (2021)
- [Autonomous Driving using Safe Reinforcement Learning by Incorporating a Regret-based Human Lane-Changing Decision Model](https://mlsanity.com/paper/1910.04803/) (2019)
- [Revisiting Modality Imbalance In Multimodal Pedestrian Detection](https://mlsanity.com/paper/2302.12589/) (2023)
- [Deep Multi-Task Networks For Occluded Pedestrian Pose Estimation](https://mlsanity.com/paper/2206.07510/) (2022)
- [Navigating Uncertainty: The Role of Short-Term Trajectory Prediction in Autonomous Vehicle Safety](https://mlsanity.com/paper/2307.05288/) (2023)

## Built on by (most cited first, 3 of 3)

- [Velocity Driven Vision: Asynchronous Sensor Fusion Birds Eye View Models for Autonomous Vehicles](https://mlsanity.com/paper/2407.16636/) (2024)
- [Addressing Diverging Training Costs using BEVRestore for High-resolution Bird's Eye View Map Construction](https://mlsanity.com/paper/2405.01016/) (2024)
- [MapsTP: HD Map Images Based Multimodal Trajectory Prediction for Automated Vehicles](https://mlsanity.com/paper/2407.05811/) (2024)

## Abstract (arXiv)

Trajectory prediction is, naturally, a key task for vehicle autonomy. While the number of traffic rules is limited, the combinations and uncertainties associated with each agent's behaviour in real-world scenarios are nearly impossible to encode. Consequently, there is a growing interest in learning-based trajectory prediction. The proposed method in this paper predicts trajectories by considering perception and trajectory prediction as a unified system. In considering them as unified tasks, we show that there is the potential to improve the performance of perception. To achieve these goals, we present BEVSeg2TP - a surround-view camera bird's-eye-view-based joint vehicle segmentation and ego vehicle trajectory prediction system for autonomous vehicles. The proposed system uses a network trained on multiple camera views. The images are transformed using several deep learning techniques to perform semantic segmentation of objects, including other vehicles, in the scene. The segmentation outputs are fused across the camera views to obtain a comprehensive representation of the surrounding vehicles from the bird's-eye-view perspective. The system further predicts the future trajectory of the ego vehicle using a spatiotemporal probabilistic network (STPN) to optimize trajectory prediction. This network leverages information from encoder-decoder transformers and joint vehicle segmentation.