# MonoLayout: Amodal scene layout from a single image

arXiv: https://arxiv.org/abs/2002.08394 · HTML: https://mlsanity.com/paper/2002.08394/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 30 of 49)

- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs](https://mlsanity.com/paper/2203.04050/) (2022)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Images](https://mlsanity.com/paper/2110.01997/) (2021)
- [KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients](https://mlsanity.com/paper/2204.13683/) (2022)
- [Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving](https://mlsanity.com/paper/2310.02251/) (2023)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [Topology Preserving Local Road Network Estimation from Single Onboard Camera Image](https://mlsanity.com/paper/2112.10155/) (2021)
- [Trust, but Verify: Cross-Modality Fusion for HD Map Change Detection](https://mlsanity.com/paper/2212.07312/) (2022)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [GitNet: Geometric Prior-based Transformation for Birds-Eye-View Segmentation](https://mlsanity.com/paper/2204.07733/) (2022)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space](https://mlsanity.com/paper/2102.09812/) (2021)
- [RoRD: Rotation-Robust Descriptors and Orthographic Views for Local Feature Matching](https://mlsanity.com/paper/2103.08573/) (2021)
- [Image Amodal Completion: A Survey](https://mlsanity.com/paper/2207.02062/) (2022)
- [Neural Groundplans: Persistent Neural Scene Representations from a Single Image](https://mlsanity.com/paper/2207.11232/) (2022)
- [Object-level Scene Deocclusion](https://mlsanity.com/paper/2406.07706/) (2024)
- [Improving Bird's Eye View Semantic Segmentation by Task Decomposition](https://mlsanity.com/paper/2404.01925/) (2024)
- [JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenes](https://mlsanity.com/paper/2207.07895/) (2022)
- [NeMO: Neural Map Growing System for Spatiotemporal Fusion in Bird's-Eye-View and BDD-Map Benchmark](https://mlsanity.com/paper/2306.04540/) (2023)
- [Pixel to Elevation: Learning to Predict Elevation Maps at Long Range using Images for Autonomous Offroad Navigation](https://mlsanity.com/paper/2401.17484/) (2024)
- [DualCross: Cross-Modality Cross-Domain Adaptation for Monocular BEV Perception](https://mlsanity.com/paper/2305.03724/) (2023)
- [Pose and Semantic Map Based Probabilistic Forecast of Vulnerable Road Users' Trajectories](https://mlsanity.com/paper/2106.02598/) (2021)
- [NVAutoNet: Fast and Accurate 360circ 3D Visual Perception For Self Driving](https://mlsanity.com/paper/2303.12976/) (2023)
- [BEV-MODNet: Monocular Camera based Bird's Eye View Moving Object Detection for Autonomous Driving](https://mlsanity.com/paper/2107.04937/) (2021)
- [Map It Anywhere (MIA): Empowering Bird's Eye View Mapping using Large-scale Public Data](https://mlsanity.com/paper/2407.08726/) (2024)

## Abstract (arXiv)

In this paper, we address the novel, highly challenging problem of estimating the layout of a complex urban driving scenario. Given a single color image captured from a driving platform, we aim to predict the bird's-eye view layout of the road and other traffic participants. The estimated layout should reason beyond what is visible in the image, and compensate for the loss of 3D information due to projection. We dub this problem amodal scene layout estimation, which involves "hallucinating" scene layout for even parts of the world that are occluded in the image. To this end, we present MonoLayout, a deep neural network for real-time amodal scene layout estimation from a single image. We represent scene layout as a multi-channel semantic occupancy grid, and leverage adversarial feature learning to hallucinate plausible completions for occluded image parts. Due to the lack of fair baseline methods, we extend several state-of-the-art approaches for road-layout estimation and vehicle occupancy estimation in bird's-eye view to the amodal setup for rigorous evaluation. By leveraging temporal sensor fusion to generate training labels, we significantly outperform current art over a number of datasets. On the KITTI and Argoverse datasets, we outperform all baselines by a significant margin. We also make all our annotations, and code publicly available. A video abstract of this paper is available https://www.youtube.com/watch?v=HcroGyo6yRQ .