# Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networks

arXiv: https://arxiv.org/abs/1804.02176 · HTML: https://mlsanity.com/paper/1804.02176/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2018.

## Built on by (most cited first, 30 of 84)

- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [VectorMapNet: End-to-end Vectorized HD Map Learning](https://mlsanity.com/paper/2206.08920/) (2022)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Images](https://mlsanity.com/paper/2110.01997/) (2021)
- [COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction](https://mlsanity.com/paper/2312.01919/) (2023)
- [PivotNet: Vectorized Pivot Learning for End-to-end HD Map Construction](https://mlsanity.com/paper/2308.16477/) (2023)
- [Neural Map Prior for Autonomous Driving](https://mlsanity.com/paper/2304.08481/) (2023)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [Learning Ego 3D Representation as Ray Tracing](https://mlsanity.com/paper/2206.04042/) (2022)
- [Online Vectorized HD Map Construction using Geometry](https://mlsanity.com/paper/2312.03341/) (2023)
- [OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environments](https://mlsanity.com/paper/2312.09243/) (2023)
- [InstaGraM: Instance-level Graph Modeling for Vectorized HD Map Learning](https://mlsanity.com/paper/2301.04470/) (2023)
- [SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation](https://mlsanity.com/paper/2211.15656/) (2022)
- [X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation](https://mlsanity.com/paper/2210.06778/) (2022)
- [EA-LSS: Edge-aware Lift-splat-shot Framework for 3D BEV Object Detection](https://mlsanity.com/paper/2303.17895/) (2023)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception](https://mlsanity.com/paper/2303.08333/) (2023)
- [TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2303.09998/) (2023)
- [RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsing](https://mlsanity.com/paper/2309.10356/) (2023)
- [BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation](https://mlsanity.com/paper/2403.11761/) (2024)
- [Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmark](https://mlsanity.com/paper/2212.08914/) (2022)
- [Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation](https://mlsanity.com/paper/2104.12861/) (2021)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [GitNet: Geometric Prior-based Transformation for Birds-Eye-View Segmentation](https://mlsanity.com/paper/2204.07733/) (2022)
- [Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation](https://mlsanity.com/paper/2207.01878/) (2022)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [MASS: Multi-Attentional Semantic Segmentation of LiDAR Data for Dense Top-View Understanding](https://mlsanity.com/paper/2107.00346/) (2021)

## Abstract (arXiv)

In this work, we research and evaluate end-to-end learning of monocular semantic-metric occupancy grid mapping from weak binocular ground truth. The network learns to predict four classes, as well as a camera to bird's eye view mapping. At the core, it utilizes a variational encoder-decoder network that encodes the front-view visual information of the driving scene and subsequently decodes it into a 2-D top-view Cartesian coordinate system. The evaluations on Cityscapes show that the end-to-end learning of semantic-metric occupancy grids outperforms the deterministic mapping approach with flat-plane assumption by more than 12% mean IoU. Furthermore, we show that the variational sampling with a relatively small embedding vector brings robustness against vehicle dynamic perturbations, and generalizability for unseen KITTI data. Our network achieves real-time inference rates of approx. 35 Hz for an input image with a resolution of 256x512 pixels and an output map with 64x64 occupancy grid cells using a Titan V GPU.