# Towards Viewpoint Robustness in Bird's Eye View Segmentation

arXiv: https://arxiv.org/abs/2309.05192 · HTML: https://mlsanity.com/paper/2309.05192/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [GeoNeRF: Generalizing NeRF with Geometry Priors](https://mlsanity.com/paper/2111.13539/) (2021)
- [Hierarchical Multi-Scale Attention for Semantic Segmentation](https://mlsanity.com/paper/2005.10821/) (2020)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video](https://mlsanity.com/paper/2205.15838/) (2022)
- [Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image](https://mlsanity.com/paper/2012.09854/) (2020)
- [AutoRF: Learning 3D Object Radiance Fields from Single View Observations](https://mlsanity.com/paper/2204.03593/) (2022)
- [A Systematic Review of Robustness in Deep Learning for Computer Vision: Mind the gap?](https://mlsanity.com/paper/2112.00639/) (2021)
- [Surface Normal Estimation of Tilted Images via Spatial Rectifier](https://mlsanity.com/paper/2007.09264/) (2020)
- [When and how CNNs generalize to out-of-distribution category-viewpoint combinations](https://mlsanity.com/paper/2007.08032/) (2020)
- [Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation](https://mlsanity.com/paper/2111.07971/) (2021)

## Built on by (most cited first, 13 of 13)

- [MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes](https://mlsanity.com/paper/2405.14475/) (2024)
- [Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap](https://mlsanity.com/paper/2403.16092/) (2024)
- [Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2410.22461/) (2024)
- [SeaBird: Segmentation in Bird's View with Dice Loss Improves Monocular 3D Detection of Large Objects](https://mlsanity.com/paper/2403.20318/) (2024)
- [Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting](https://mlsanity.com/paper/2404.06700/) (2024)
- [CHARM3R: Towards Unseen Camera Height Robust Monocular 3D Detector](https://mlsanity.com/paper/2508.11185/) (2025)
- [VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2510.23205/) (2025)
- [PCT: Perspective Cue Training Framework for Multi-Camera BEV Segmentation](https://mlsanity.com/paper/2403.12530/) (2024)
- [When Cars meet Drones: Hyperbolic Federated Learning for Source-Free Domain Adaptation in Adverse Weather](https://mlsanity.com/paper/2403.13762/) (2024)
- [MapGS: Generalizable Pretraining and Data Augmentation for Online Mapping via Novel View Synthesis](https://mlsanity.com/paper/2501.06660/) (2025)
- [ViewpointDepth: A New Dataset for Monocular Depth Estimation Under Viewpoint Shifts](https://mlsanity.com/paper/2409.17851/) (2024)
- [Cross-view Domain Generalization via Geometric Consistency for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2602.14525/) (2026)
- [Understanding Cross-Rig Generalization in Automotive Perception: a Multi-Rig Benchmark and Rig Variation Metrics](https://mlsanity.com/paper/2606.27554/) (2026)

## Abstract (arXiv)

Autonomous vehicles (AV) require that neural networks used for perception be robust to different viewpoints if they are to be deployed across many types of vehicles without the repeated cost of data collection and labeling for each. AV companies typically focus on collecting data from diverse scenarios and locations, but not camera rig configurations, due to cost. As a result, only a small number of rig variations exist across most fleets. In this paper, we study how AV perception models are affected by changes in camera viewpoint and propose a way to scale them across vehicle types without repeated data collection and labeling. Using bird's eye view (BEV) segmentation as a motivating task, we find through extensive experiments that existing perception models are surprisingly sensitive to changes in camera viewpoint. When trained with data from one camera rig, small changes to pitch, yaw, depth, or height of the camera at inference time lead to large drops in performance. We introduce a technique for novel view synthesis and use it to transform collected data to the viewpoint of target rigs, allowing us to train BEV segmentation models for diverse target rigs without any additional data collection or labeling cost. To analyze the impact of viewpoint changes, we leverage synthetic data to mitigate other gaps (content, ISP, etc). Our approach is then trained on real data and evaluated on synthetic data, enabling evaluation on diverse target rigs. We release all data for use in future work. Our method is able to recover an average of 14.7% of the IoU that is otherwise lost when deploying to new rigs.