# Learning to Look around Objects for Top-View Representations of Outdoor Scenes

arXiv: https://arxiv.org/abs/1803.10870 · HTML: https://mlsanity.com/paper/1803.10870/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2018.

## Built on by (most cited first, 30 of 37)

- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?](https://mlsanity.com/paper/2206.07959/) (2022)
- [BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs](https://mlsanity.com/paper/2203.04050/) (2022)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [InstaGraM: Instance-level Graph Modeling for Vectorized HD Map Learning](https://mlsanity.com/paper/2301.04470/) (2023)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmark](https://mlsanity.com/paper/2212.08914/) (2022)
- [GitNet: Geometric Prior-based Transformation for Birds-Eye-View Segmentation](https://mlsanity.com/paper/2204.07733/) (2022)
- [Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation](https://mlsanity.com/paper/2207.01878/) (2022)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [HFT: Lifting Perspective Representations via Hybrid Feature Transformation](https://mlsanity.com/paper/2204.05068/) (2022)
- [Multi-Agent Variational Occlusion Inference Using People as Sensors](https://mlsanity.com/paper/2109.02173/) (2021)
- [Rethinking Amodal Video Segmentation from Learning Supervised Signals with Object-centric Representation](https://mlsanity.com/paper/2309.13248/) (2023)
- [BEV-CV: Birds-Eye-View Transform for Cross-View Geo-Localisation](https://mlsanity.com/paper/2312.15363/) (2023)
- [Causal Imitative Model for Autonomous Driving](https://mlsanity.com/paper/2112.03908/) (2021)
- [A Cross-Scale Hierarchical Transformer with Correspondence-Augmented Attention for inferring Bird's-Eye-View Semantic Segmentation](https://mlsanity.com/paper/2304.03650/) (2023)
- [Real-time 3D semantic occupancy prediction for autonomous vehicles using memory-efficient sparse convolution](https://mlsanity.com/paper/2403.08748/) (2024)
- [NVAutoNet: Fast and Accurate 360circ 3D Visual Perception For Self Driving](https://mlsanity.com/paper/2303.12976/) (2023)
- [Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models](https://mlsanity.com/paper/2210.14584/) (2022)
- ["The Pedestrian next to the Lamppost" Adaptive Object Graphs for Better Instantaneous Mapping](https://mlsanity.com/paper/2204.02944/) (2022)
- [Monocular BEV Perception of Road Scenes via Front-to-Top View Projection](https://mlsanity.com/paper/2211.08144/) (2022)
- [Learning Road Scene-level Representations via Semantic Region Prediction](https://mlsanity.com/paper/2301.00714/) (2023)
- [Predictive World Models from Real-World Partial Observations](https://mlsanity.com/paper/2301.04783/) (2023)
- [Weakly But Deeply Supervised Occlusion-Reasoned Parametric Road Layouts](https://mlsanity.com/paper/2104.06730/) (2021)
- [WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion](https://mlsanity.com/paper/2403.19022/) (2024)
- [nuScenes Revisited: Progress and Challenges in Autonomous Driving](https://mlsanity.com/paper/2512.02448/) (2025)
- [Monocular Multi-Layer Layout Estimation for Warehouse Racks](https://mlsanity.com/paper/2103.09174/) (2021)
- [Foreground-guided Facial Inpainting with Fidelity Preservation](https://mlsanity.com/paper/2105.03342/) (2021)
- [SBEVNet: End-to-End Deep Stereo Layout Estimation](https://mlsanity.com/paper/2105.11705/) (2021)

## Abstract (arXiv)

Given a single RGB image of a complex outdoor road scene in the perspective view, we address the novel problem of estimating an occlusion-reasoned semantic scene layout in the top-view. This challenging problem not only requires an accurate understanding of both the 3D geometry and the semantics of the visible scene, but also of occluded areas. We propose a convolutional neural network that learns to predict occluded portions of the scene layout by looking around foreground objects like cars or pedestrians. But instead of hallucinating RGB values, we show that directly predicting the semantics and depths in the occluded areas enables a better transformation into the top-view. We further show that this initial top-view representation can be significantly enhanced by learning priors and rules about typical road layouts from simulated or, if available, map data. Crucially, training our model does not require costly or subjective human annotations for occluded areas or the top-view, but rather uses readily available annotations for standard semantic segmentation. We extensively evaluate and analyze our approach on the KITTI and Cityscapes data sets.