# ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop

arXiv: https://arxiv.org/abs/2603.02481 · HTML: https://mlsanity.com/paper/2603.02481/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Builds on (most cited first, 17 of 17)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [Cross Modal Transformer: Towards Fast and Robust 3D Object Detection](https://mlsanity.com/paper/2301.01283/) (2023)
- [Behind the Curtain: Learning Occluded Shapes for 3D Object Detection](https://mlsanity.com/paper/2112.02205/) (2021)
- [Temporal Enhanced Training of Multi-view 3D Object Detector via Historical Object Prediction](https://mlsanity.com/paper/2304.00967/) (2023)
- [A Benchmark for Lidar Sensors in Fog: Is Detection Breaking Down?](https://mlsanity.com/paper/1912.03251/) (2019)
- [BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregation](https://mlsanity.com/paper/2303.17099/) (2023)
- [UniBEV: Multi-modal 3D Object Detection with Uniform BEV Encoders for Robustness against Missing Sensor Modalities](https://mlsanity.com/paper/2309.14516/) (2023)
- [SparseLIF: High-Performance Sparse LiDAR-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2403.07284/) (2024)
- [Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness](https://mlsanity.com/paper/2503.18445/) (2025)
- [Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather](https://mlsanity.com/paper/1902.08913/) (2019)
- [Robust Multimodal 3D Object Detection via Modality-Agnostic Decoding and Proximity-based Modality Ensemble](https://mlsanity.com/paper/2407.19156/) (2024)

## Abstract (arXiv)

Multi-modal 3D object detection is pivotal for autonomous driving, integrating complementary sensors like LiDAR and cameras. However, its real-world reliability is challenged by transient data interruptions and missing, where modalities can momentarily drop due to hardware glitches, adverse weather, or occlusions. This poses a critical risk, especially during a simultaneous modality drop, where the vehicle is momentarily blind. To address this problem, we introduce ModalPatch, the first plug-and-play module designed to enable robust detection under arbitrary modality-drop scenarios. Without requiring architectural changes or retraining, ModalPatch can be seamlessly integrated into diverse detection frameworks. Technically, ModalPatch leverages the temporal nature of sensor data for perceptual continuity, using a history-based module to predict and compensate for transiently unavailable features. To improve the fidelity of the predicted features, we further introduce an uncertainty-guided cross-modality fusion strategy that dynamically estimates the reliability of compensated features, suppressing biased signals while reinforcing informative ones. Extensive experiments show that ModalPatch consistently enhances both robustness and accuracy of state-of-the-art 3D object detectors under diverse modality-drop conditions.