# Online Lane Graph Extraction from Onboard Video

arXiv: https://arxiv.org/abs/2304.00930 · HTML: https://mlsanity.com/paper/2304.00930/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 43 of its 52 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 30 of 43)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [HDMapNet: An Online HD Map Construction and Evaluation Framework](https://mlsanity.com/paper/2107.06307/) (2021)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [Cross-view Semantic Segmentation for Sensing Surroundings](https://mlsanity.com/paper/1906.03560/) (2019)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Towards End-to-End Lane Detection: an Instance Segmentation Approach](https://mlsanity.com/paper/1802.05591/) (2018)
- [Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++](https://mlsanity.com/paper/1803.09693/) (2018)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [3D-LaneNet: End-to-End 3D Multiple Lane Detection](https://mlsanity.com/paper/1811.10203/) (2018)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Images](https://mlsanity.com/paper/2110.01997/) (2021)
- [HDNET: Exploiting HD Maps for 3D Object Detection](https://mlsanity.com/paper/2012.11704/) (2020)
- [Key Points Estimation and Point Instance Segmentation Approach for Lane Detection](https://mlsanity.com/paper/2002.06604/) (2020)
- [FISHING Net: Future Inference of Semantic Heatmaps In Grids](https://mlsanity.com/paper/2006.09917/) (2020)
- [DAGMapper: Learning to Map by Discovering Lane Topology](https://mlsanity.com/paper/2012.12377/) (2020)
- [Long-Range Transformers for Dynamic Spatiotemporal Forecasting](https://mlsanity.com/paper/2109.12218/) (2021)
- [Hierarchical Recurrent Attention Networks for Structured Online Maps](https://mlsanity.com/paper/2012.12314/) (2020)
- [End-to-end Lane Detection through Differentiable Least-Squares Fitting](https://mlsanity.com/paper/1902.00293/) (2019)

## Abstract (arXiv)

Autonomous driving requires a structured understanding of the surrounding road network to navigate. One of the most common and useful representation of such an understanding is done in the form of BEV lane graphs. In this work, we use the video stream from an onboard camera for online extraction of the surrounding's lane graph. Using video, instead of a single image, as input poses both benefits and challenges in terms of combining the information from different timesteps. We study the emerged challenges using three different approaches. The first approach is a post-processing step that is capable of merging single frame lane graph estimates into a unified lane graph. The second approach uses the spatialtemporal embeddings in the transformer to enable the network to discover the best temporal aggregation strategy. Finally, the third, and the proposed method, is an early temporal aggregation through explicit BEV projection and alignment of framewise features. A single model of this proposed simple, yet effective, method can process any number of images, including one, to produce accurate lane graphs. The experiments on the Nuscenes and Argoverse datasets show the validity of all the approaches while highlighting the superiority of the proposed method. The code will be made public.