# Learning from Maps: Visual Common Sense for Autonomous Driving

arXiv: https://arxiv.org/abs/1611.08583 · HTML: https://mlsanity.com/paper/1611.08583/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2016.

## Built on by (most cited first, 4 of 4)

- [ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps](https://mlsanity.com/paper/2505.18675/) (2025)
- [Image-based Geolocalization by Ground-to-2.5D Map Matching](https://mlsanity.com/paper/2308.05993/) (2023)
- [Weakly But Deeply Supervised Occlusion-Reasoned Parametric Road Layouts](https://mlsanity.com/paper/2104.06730/) (2021)
- [Preconditioned Visual Language Inference with Weak Supervision](https://mlsanity.com/paper/2306.01753/) (2023)

## Abstract (arXiv)

Today's autonomous vehicles rely extensively on high-definition 3D maps to navigate the environment. While this approach works well when these maps are completely up-to-date, safe autonomous vehicles must be able to corroborate the map's information via a real time sensor-based system. Our goal in this work is to develop a model for road layout inference given imagery from on-board cameras, without any reliance on high-definition maps. However, no sufficient dataset for training such a model exists. Here, we leverage the availability of standard navigation maps and corresponding street view images to construct an automatically labeled, large-scale dataset for this complex scene understanding problem. By matching road vectors and metadata from navigation maps with Google Street View images, we can assign ground truth road layout attributes (e.g., distance to an intersection, one-way vs. two-way street) to the images. We then train deep convolutional networks to predict these road layout attributes given a single monocular RGB image. Experimental evaluation demonstrates that our model learns to correctly infer the road attributes using only panoramas captured by car-mounted cameras as input. Additionally, our results indicate that this method may be suitable to the novel application of recommending safety improvements to infrastructure (e.g., suggesting an alternative speed limit for a street).