# A Black-Box Evaluation Framework for Semantic Robustness in Bird's Eye View Detection

arXiv: https://arxiv.org/abs/2412.13913 · HTML: https://mlsanity.com/paper/2412.13913/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [Model-Based Imitation Learning for Urban Driving](https://mlsanity.com/paper/2210.07729/) (2022)
- [Learning from All Vehicles](https://mlsanity.com/paper/2203.11934/) (2022)
- [Robo3D: Towards Robust and Reliable 3D Perception against Corruptions](https://mlsanity.com/paper/2303.17597/) (2023)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [TSS: Transformation-Specific Smoothing for Robustness Certification](https://mlsanity.com/paper/2002.12398/) (2020)
- [RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions](https://mlsanity.com/paper/2310.15171/) (2023)
- [RoboBEV: Towards Robust Bird's Eye View Perception under Corruptions](https://mlsanity.com/paper/2304.06719/) (2023)
- [Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Driving](https://mlsanity.com/paper/2303.17297/) (2023)
- [Robustness of Object Detectors in Degrading Weather Conditions](https://mlsanity.com/paper/2106.08795/) (2021)
- [Evaluating Adversarial Attacks on Driving Safety in Vision-Based Autonomous Vehicles](https://mlsanity.com/paper/2108.02940/) (2021)
- [PRoA: A Probabilistic Robustness Assessment against Functional Perturbations](https://mlsanity.com/paper/2207.02036/) (2022)
- [Bayesian Optimization with Exponential Convergence](https://mlsanity.com/paper/1604.01348/) (2016)
- [Identifying Model Weakness with Adversarial Examiner](https://mlsanity.com/paper/1911.11230/) (2019)
- [DIMBA: Discretely Masked Black-Box Attack in Single Object Tracking](https://mlsanity.com/paper/2207.08044/) (2022)
- [Towards Verifying the Geometric Robustness of Large-scale Neural Networks](https://mlsanity.com/paper/2301.12456/) (2023)
- [Large Norms of CNN Layers Do Not Hurt Adversarial Robustness](https://mlsanity.com/paper/2009.08435/) (2020)
- [Sensor Adversarial Traits: Analyzing Robustness of 3D Object Detection Sensor Fusion Models](https://mlsanity.com/paper/2109.06363/) (2021)

## Built on by (most cited first, 3 of 3)

- [ProGRank: Probe-Gradient Reranking to Defend Dense-Retriever RAG from Corpus Poisoning](https://mlsanity.com/paper/2603.22934/) (2026)
- [IoUCert: Robustness Verification for Anchor-based Object Detectors](https://mlsanity.com/paper/2603.03043/) (2026)
- [RESBev: Making BEV Perception More Robust](https://mlsanity.com/paper/2603.09529/) (2026)

## Abstract (arXiv)

Camera-based Bird's Eye View (BEV) perception models receive increasing attention for their crucial role in autonomous driving, a domain where concerns about the robustness and reliability of deep learning have been raised. While only a few works have investigated the effects of randomly generated semantic perturbations, aka natural corruptions, on the multi-view BEV detection task, we develop a black-box robustness evaluation framework that adversarially optimises three common semantic perturbations: geometric transformation, colour shifting, and motion blur, to deceive BEV models, serving as the first approach in this emerging field. To address the challenge posed by optimising the semantic perturbation, we design a smoothed, distance-based surrogate function to replace the mAP metric and introduce SimpleDIRECT, a deterministic optimisation algorithm that utilises observed slopes to guide the optimisation process. By comparing with randomised perturbation and two optimisation baselines, we demonstrate the effectiveness of the proposed framework. Additionally, we provide a benchmark on the semantic robustness of ten recent BEV models. The results reveal that PolarFormer, which emphasises geometric information from multi-view images, exhibits the highest robustness, whereas BEVDet is fully compromised, with its precision reduced to zero.