# A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye View

arXiv: https://arxiv.org/abs/2005.04078 · HTML: https://mlsanity.com/paper/2005.04078/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 30 of 72)

- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs](https://mlsanity.com/paper/2203.04050/) (2022)
- [Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe](https://mlsanity.com/paper/2209.05324/) (2022)
- [End-to-End Vectorized HD-map Construction with Piecewise Bezier Curve](https://mlsanity.com/paper/2306.09700/) (2023)
- [Uncertainty-aware Vision-based Metric Cross-view Geolocalization](https://mlsanity.com/paper/2211.12145/) (2022)
- [MGMap: Mask-Guided Learning for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2404.00876/) (2024)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [InstaGraM: Instance-level Graph Modeling for Vectorized HD Map Learning](https://mlsanity.com/paper/2301.04470/) (2023)
- [Cross-view image geo-localization with Panorama-BEV Co-Retrieval Network](https://mlsanity.com/paper/2408.05475/) (2024)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation](https://mlsanity.com/paper/2206.13294/) (2022)
- [MatrixVT: Efficient Multi-Camera to BEV Transformation for 3D Perception](https://mlsanity.com/paper/2211.10593/) (2022)
- [Mask2Map: Vectorized HD Map Construction Using Bird's Eye View Segmentation Masks](https://mlsanity.com/paper/2407.13517/) (2024)
- [Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation](https://mlsanity.com/paper/2104.12861/) (2021)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [GitNet: Geometric Prior-based Transformation for Birds-Eye-View Segmentation](https://mlsanity.com/paper/2204.07733/) (2022)
- [Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation](https://mlsanity.com/paper/2207.01878/) (2022)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [Monocular 3D Vehicle Detection Using Uncalibrated Traffic Cameras through Homography](https://mlsanity.com/paper/2103.15293/) (2021)
- [DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention](https://mlsanity.com/paper/2212.07849/) (2022)
- [RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation](https://mlsanity.com/paper/2406.18977/) (2024)
- [MV-Map: Offboard HD-Map Generation with Multi-view Consistency](https://mlsanity.com/paper/2305.08851/) (2023)
- [SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation](https://mlsanity.com/paper/2404.02638/) (2024)
- [MapLocNet: Coarse-to-Fine Feature Registration for Visual Re-Localization in Navigation Maps](https://mlsanity.com/paper/2407.08561/) (2024)
- [Neural Groundplans: Persistent Neural Scene Representations from a Single Image](https://mlsanity.com/paper/2207.11232/) (2022)
- [ScalableMap: Scalable Map Learning for Online Long-Range Vectorized HD Map Construction](https://mlsanity.com/paper/2310.13378/) (2023)
- [Bi-Mapper: Holistic BEV Semantic Mapping for Autonomous Driving](https://mlsanity.com/paper/2305.04205/) (2023)
- [Improving Bird's Eye View Semantic Segmentation by Task Decomposition](https://mlsanity.com/paper/2404.01925/) (2024)

## Abstract (arXiv)

Accurate environment perception is essential for automated driving. When using monocular cameras, the distance estimation of elements in the environment poses a major challenge. Distances can be more easily estimated when the camera perspective is transformed to a bird's eye view (BEV). For flat surfaces, Inverse Perspective Mapping (IPM) can accurately transform images to a BEV. Three-dimensional objects such as vehicles and vulnerable road users are distorted by this transformation making it difficult to estimate their position relative to the sensor. This paper describes a methodology to obtain a corrected 360deg BEV image given images from multiple vehicle-mounted cameras. The corrected BEV image is segmented into semantic classes and includes a prediction of occluded areas. The neural network approach does not rely on manually labeled data, but is trained on a synthetic dataset in such a way that it generalizes well to real-world data. By using semantically segmented images as input, we reduce the reality gap between simulated and real-world data and are able to show that our method can be successfully applied in the real world. Extensive experiments conducted on the synthetic data demonstrate the superiority of our approach compared to IPM. Source code and datasets are available at https://github.com/ika-rwth-aachen/Cam2BEV