# Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

arXiv: https://arxiv.org/abs/2108.03227 · HTML: https://mlsanity.com/paper/2108.03227/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 6 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [Cross-view Semantic Segmentation for Sensing Surroundings](https://mlsanity.com/paper/1906.03560/) (2019)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [SSAP: Single-Shot Instance Segmentation With Affinity Pyramid](https://mlsanity.com/paper/1909.01616/) (2019)
- [Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networks](https://mlsanity.com/paper/1804.02176/) (2018)
- [A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye View](https://mlsanity.com/paper/2005.04078/) (2020)
- [MOPT: Multi-Object Panoptic Tracking](https://mlsanity.com/paper/2004.08189/) (2020)
- [MonoLayout: Amodal scene layout from a single image](https://mlsanity.com/paper/2002.08394/) (2020)
- [Weakly- and Semi-Supervised Panoptic Segmentation](https://mlsanity.com/paper/1808.03575/) (2018)
- [Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer](https://mlsanity.com/paper/1511.03240/) (2015)
- [BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud](https://mlsanity.com/paper/2006.11436/) (2020)
- [Learning to Look around Objects for Top-View Representations of Outdoor Scenes](https://mlsanity.com/paper/1803.10870/) (2018)
- [A Geometric Approach to Obtain a Bird's Eye View from an Image](https://mlsanity.com/paper/1905.02231/) (2019)
- [Learning to Fuse Things and Stuff](https://mlsanity.com/paper/1812.01192/) (2018)
- [Generative Adversarial Frontal View to Bird View Synthesis](https://mlsanity.com/paper/1808.00327/) (2018)
- [Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing](https://mlsanity.com/paper/1808.06887/) (2018)
- [Learning to Map Vehicles into Bird's Eye View](https://mlsanity.com/paper/1706.08442/) (2017)
- [The Right (Angled) Perspective: Improving the Understanding of Road Scenes Using Boosted Inverse Perspective Mapping](https://mlsanity.com/paper/1812.00913/) (2018)

## Built on by (most cited first, 30 of 48)

- [M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation](https://mlsanity.com/paper/2204.05088/) (2022)
- [Model-Based Imitation Learning for Urban Driving](https://mlsanity.com/paper/2210.07729/) (2022)
- [Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?](https://mlsanity.com/paper/2206.07959/) (2022)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe](https://mlsanity.com/paper/2209.05324/) (2022)
- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [Segment Anything in 3D with Radiance Fields](https://mlsanity.com/paper/2304.12308/) (2023)
- [Amodal Panoptic Segmentation](https://mlsanity.com/paper/2202.11542/) (2022)
- [X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation](https://mlsanity.com/paper/2210.06778/) (2022)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation](https://mlsanity.com/paper/2403.11761/) (2024)
- [Perceiving the Invisible: Proposal-Free Amodal Panoptic Segmentation](https://mlsanity.com/paper/2205.14637/) (2022)
- [MapPrior: Bird's-Eye View Map Layout Estimation with Generative Models](https://mlsanity.com/paper/2308.12963/) (2023)
- [MapFusion: A Novel BEV Feature Fusion Network for Multi-modal Map Construction](https://mlsanity.com/paper/2502.04377/) (2025)
- [Learning and Aggregating Lane Graphs for Urban Automated Driving](https://mlsanity.com/paper/2302.06175/) (2023)
- [Fast-BEV: Towards Real-time On-vehicle Bird's-Eye View Perception](https://mlsanity.com/paper/2301.07870/) (2023)
- [CoDEPS: Online Continual Learning for Depth Estimation and Panoptic Segmentation](https://mlsanity.com/paper/2303.10147/) (2023)
- [Uncertainty-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.14554/) (2022)
- [Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning](https://mlsanity.com/paper/2206.10711/) (2022)
- [Bi-Mapper: Holistic BEV Semantic Mapping for Autonomous Driving](https://mlsanity.com/paper/2305.04205/) (2023)
- [U-BEV: Height-aware Bird's-Eye-View Segmentation and Neural Map-based Relocalization](https://mlsanity.com/paper/2310.13766/) (2023)
- [Improving Bird's Eye View Semantic Segmentation by Task Decomposition](https://mlsanity.com/paper/2404.01925/) (2024)
- [SeaBird: Segmentation in Bird's View with Dice Loss Improves Monocular 3D Detection of Large Objects](https://mlsanity.com/paper/2403.20318/) (2024)
- [BLOS-BEV: Navigation Map Enhanced Lane Segmentation Network, Beyond Line of Sight](https://mlsanity.com/paper/2407.08526/) (2024)
- [NeMO: Neural Map Growing System for Spatiotemporal Fusion in Bird's-Eye-View and BDD-Map Benchmark](https://mlsanity.com/paper/2306.04540/) (2023)
- [PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes](https://mlsanity.com/paper/2309.10815/) (2023)
- [Self-Supervised Representation Learning from Temporal Ordering of Automated Driving Sequences](https://mlsanity.com/paper/2302.09043/) (2023)
- [Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation](https://mlsanity.com/paper/2308.14525/) (2023)
- [ViT-BEVSeg: A Hierarchical Transformer Network for Monocular Birds-Eye-View Segmentation](https://mlsanity.com/paper/2205.15667/) (2022)
- [Contrastive Object Detection Using Knowledge Graph Embeddings](https://mlsanity.com/paper/2112.11366/) (2021)

## Abstract (arXiv)

Bird's-Eye-View (BEV) maps have emerged as one of the most powerful representations for scene understanding due to their ability to provide rich spatial context while being easy to interpret and process. Such maps have found use in many real-world tasks that extensively rely on accurate scene segmentation as well as object instance identification in the BEV space for their operation. However, existing segmentation algorithms only predict the semantics in the BEV space, which limits their use in applications where the notion of object instances is also critical. In this work, we present the first BEV panoptic segmentation approach for directly predicting dense panoptic segmentation maps in the BEV, given a single monocular image in the frontal view (FV). Our architecture follows the top-down paradigm and incorporates a novel dense transformer module consisting of two distinct transformers that learn to independently map vertical and flat regions in the input image from the FV to the BEV. Additionally, we derive a mathematical formulation for the sensitivity of the FV-BEV transformation which allows us to intelligently weight pixels in the BEV space to account for the varying descriptiveness across the FV image. Extensive evaluations on the KITTI-360 and nuScenes datasets demonstrate that our approach exceeds the state-of-the-art in the PQ metric by 3.61 pp and 4.93 pp respectively.