# Self-Supervised Road Layout Parsing with Graph Auto-Encoding

arXiv: https://arxiv.org/abs/2203.11000 · HTML: https://mlsanity.com/paper/2203.11000/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networks](https://mlsanity.com/paper/1804.02176/) (2018)
- [Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation](https://mlsanity.com/paper/1806.11538/) (2018)
- [Attentive Relational Networks for Mapping Images to Scene Graphs](https://mlsanity.com/paper/1811.10696/) (2018)
- [Fully Convolutional Scene Graph Generation](https://mlsanity.com/paper/2103.16083/) (2021)
- [Part-aware Panoptic Segmentation](https://mlsanity.com/paper/2106.06351/) (2021)
- [MonoLayout: Amodal scene layout from a single image](https://mlsanity.com/paper/2002.08394/) (2020)
- [Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction](https://mlsanity.com/paper/1802.05451/) (2018)
- [Learning to Look around Objects for Top-View Representations of Outdoor Scenes](https://mlsanity.com/paper/1803.10870/) (2018)
- [A Parametric Top-View Representation of Complex Road Scenes](https://mlsanity.com/paper/1812.06152/) (2018)
- [Image-Graph-Image Translation via Auto-Encoding](https://mlsanity.com/paper/2012.05975/) (2020)

## Abstract (arXiv)

Aiming for higher-level scene understanding, this work presents a neural network approach that takes a road-layout map in bird's-eye-view as input, and predicts a human-interpretable graph that represents the road's topological layout. Our approach elevates the understanding of road layouts from pixel level to the level of graphs. To achieve this goal, an image-graph-image auto-encoder is utilized. The network is designed to learn to regress the graph representation at its auto-encoder bottleneck. This learning is self-supervised by an image reconstruction loss, without needing any external manual annotations. We create a synthetic dataset containing common road layout patterns and use it for training of the auto-encoder in addition to the real-world Argoverse dataset. By using this additional synthetic dataset, which conceptually captures human knowledge of road layouts and makes this available to the network for training, we are able to stabilize and further improve the performance of topological road layout understanding on the real-world Argoverse dataset. The evaluation shows that our approach exhibits comparable performance to a strong fully-supervised baseline.