# UniBEV: Multi-modal 3D Object Detection with Uniform BEV Encoders for Robustness against Missing Sensor Modalities

arXiv: https://arxiv.org/abs/2309.14516 · HTML: https://mlsanity.com/paper/2309.14516/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 36 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection](https://mlsanity.com/paper/1908.09492/) (2019)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [FUTR3D: A Unified Sensor Fusion Framework for 3D Detection](https://mlsanity.com/paper/2203.10642/) (2022)
- [DeepInteraction: 3D Object Detection via Modality Interaction](https://mlsanity.com/paper/2208.11112/) (2022)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [HDNET: Exploiting HD Maps for 3D Object Detection](https://mlsanity.com/paper/2012.11704/) (2020)
- [CBNet: A Composite Backbone Network Architecture for Object Detection](https://mlsanity.com/paper/2107.00420/) (2021)
- [AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2201.06493/) (2022)
- [Object DGCNN: 3D Object Detection using Dynamic Graphs](https://mlsanity.com/paper/2110.06923/) (2021)
- [X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation](https://mlsanity.com/paper/2210.06778/) (2022)

## Built on by (most cited first, 29 of 29)

- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)
- [FG2: Fine-Grained Cross-View Localization by Fine-Grained Feature Matching](https://mlsanity.com/paper/2503.18725/) (2025)
- [DPFT: Dual Perspective Fusion Transformer for Camera-Radar-based Object Detection](https://mlsanity.com/paper/2404.03015/) (2024)
- [Resilient Sensor Fusion under Adverse Sensor Failures via Multi-Modal Expert Fusion](https://mlsanity.com/paper/2503.19776/) (2025)
- [SafeMap: Robust HD Map Construction from Incomplete Observations](https://mlsanity.com/paper/2507.00861/) (2025)
- [SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weather](https://mlsanity.com/paper/2508.16408/) (2025)
- [Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation](https://mlsanity.com/paper/2401.16923/) (2024)
- [VoteFlow: Enforcing Local Rigidity in Self-Supervised Scene Flow](https://mlsanity.com/paper/2503.22328/) (2025)
- [Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies](https://mlsanity.com/paper/2603.16952/) (2026)
- [SparseFusion: Efficient Sparse Multi-Modal Fusion Framework for Long-Range 3D Perception](https://mlsanity.com/paper/2403.10036/) (2024)
- [MCE: Towards a General Framework for Handling Missing Modalities under Imbalanced Missing Rates](https://mlsanity.com/paper/2510.10534/) (2025)
- [PromptDet: A Lightweight 3D Object Detection Framework with LiDAR Prompts](https://mlsanity.com/paper/2412.12460/) (2024)
- [AutoHood3D: A Multi-Modal Benchmark for Automotive Hood Design and Fluid-Structure Interaction](https://mlsanity.com/paper/2511.05596/) (2025)
- [Post Fusion Bird's Eye View Feature Stabilization for Robust Multimodal 3D Detection](https://mlsanity.com/paper/2603.05623/) (2026)
- [CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2603.23276/) (2026)
- [From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving](https://mlsanity.com/paper/2509.23641/) (2025)
- [AsyncBEV: Cross-modal Flow Alignment in Asynchronous 3D Object Detection](https://mlsanity.com/paper/2601.12994/) (2026)
- [4DRC-OCC: Robust Semantic Occupancy Prediction Through Fusion of 4D Radar and Camera](https://mlsanity.com/paper/2603.07794/) (2026)
- [Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion](https://mlsanity.com/paper/2508.18734/) (2025)
- [LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptation](https://mlsanity.com/paper/2512.20217/) (2025)
- [SiMO: Single-Modality-Operable Multimodal Collaborative Perception](https://mlsanity.com/paper/2603.08240/) (2026)
- [M2-Occ: Resilient 3D Semantic Occupancy Prediction for Autonomous Driving with Incomplete Camera Inputs](https://mlsanity.com/paper/2603.09737/) (2026)
- [End-to-End 3-D Spatiotemporal Perception with Multimodal Fusion and V2X Collaboration](https://mlsanity.com/paper/2512.21831/) (2025)
- [ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop](https://mlsanity.com/paper/2603.02481/) (2026)
- [SB-BEVFusion: Enhancing the Robustness against Sensor Malfunction and Corruptions](https://mlsanity.com/paper/2605.11799/) (2026)
- [Can BEV Perception Gracefully Degrade under Sensor Failures?](https://mlsanity.com/paper/2605.30983/) (2026)
- [Fusing Satellite Imagery and Planimetric Maps for Cross-View Localization](https://mlsanity.com/paper/2606.10166/) (2026)
- [Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation](https://mlsanity.com/paper/2608.24366/) (2026)
- [SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts](https://mlsanity.com/paper/2610.15745/) (2026)

## Abstract (arXiv)

Multi-sensor object detection is an active research topic in automated driving, but the robustness of such detection models against missing sensor input (modality missing), e.g., due to a sudden sensor failure, is a critical problem which remains under-studied. In this work, we propose UniBEV, an end-to-end multi-modal 3D object detection framework designed for robustness against missing modalities: UniBEV can operate on LiDAR plus camera input, but also on LiDAR-only or camera-only input without retraining. To facilitate its detector head to handle different input combinations, UniBEV aims to create well-aligned Bird's Eye View (BEV) feature maps from each available modality. Unlike prior BEV-based multi-modal detection methods, all sensor modalities follow a uniform approach to resample features from the native sensor coordinate systems to the BEV features. We furthermore investigate the robustness of various fusion strategies w.r.t. missing modalities: the commonly used feature concatenation, but also channel-wise averaging, and a generalization to weighted averaging termed Channel Normalized Weights. To validate its effectiveness, we compare UniBEV to state-of-the-art BEVFusion and MetaBEV on nuScenes over all sensor input combinations. In this setting, UniBEV achieves 52.5 % mAP on average over all input combinations, significantly improving over the baselines (43.5 % mAP on average for BEVFusion, 48.7 % mAP on average for MetaBEV). An ablation study shows the robustness benefits of fusing by weighted averaging over regular concatenation, and of sharing queries between the BEV encoders of each modality. Our code is available at https://github.com/tudelft-iv/UniBEV.