# RoboFusion: Towards Robust Multi-Modal 3D Object Detection via SAM

arXiv: https://arxiv.org/abs/2401.03907 · HTML: https://mlsanity.com/paper/2401.03907/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Builds on (most cited first, 30 of 39)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [FUTR3D: A Unified Sensor Fusion Framework for 3D Detection](https://mlsanity.com/paper/2203.10642/) (2022)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)
- [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection](https://mlsanity.com/paper/2007.08856/) (2020)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [DeepInteraction: 3D Object Detection via Modality Interaction](https://mlsanity.com/paper/2208.11112/) (2022)
- [Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving](https://mlsanity.com/paper/2303.11040/) (2023)
- [Unsupervised Domain Adaptation of Object Detectors: A Survey](https://mlsanity.com/paper/2105.13502/) (2021)
- [SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2304.14340/) (2023)
- [LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion](https://mlsanity.com/paper/2303.03595/) (2023)
- [SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds](https://mlsanity.com/paper/2004.02774/) (2020)
- [Multi-Modal 3D Object Detection in Autonomous Driving: a Survey](https://mlsanity.com/paper/2106.12735/) (2021)
- [Wavelet-Based Dual-Branch Network for Image Demoireing](https://mlsanity.com/paper/2007.07173/) (2020)
- [Voxel Field Fusion for 3D Object Detection](https://mlsanity.com/paper/2205.15938/) (2022)

## Built on by (most cited first, 20 of 20)

- [GraphBEV: Towards Robust BEV Feature Alignment for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2403.11848/) (2024)
- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [V2X-Radar: A Multi-modal Dataset with 4D Radar for Cooperative Perception](https://mlsanity.com/paper/2411.10962/) (2024)
- [ContrastAlign: Toward Robust BEV Feature Alignment via Contrastive Learning for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2405.16873/) (2024)
- [DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation](https://mlsanity.com/paper/2503.11122/) (2025)
- [RoadBEV: Road Surface Reconstruction in Bird's Eye View](https://mlsanity.com/paper/2404.06605/) (2024)
- [SparseDet: A Simple and Effective Framework for Fully Sparse LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2406.10907/) (2024)
- [SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection](https://mlsanity.com/paper/2410.11358/) (2024)
- [TiGDistill-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning Distillation](https://mlsanity.com/paper/2412.20911/) (2024)
- [Towards Accurate and Efficient 3D Object Detection for Autonomous Driving: A Mixture of Experts Computing System on Edge](https://mlsanity.com/paper/2507.04123/) (2025)
- [DGFusion: Dual-guided Fusion for Robust Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2511.10035/) (2025)
- [AVS-Net: Point Sampling with Adaptive Voxel Size for 3D Scene Understanding](https://mlsanity.com/paper/2402.17521/) (2024)
- [Look Before You Fuse: 2D-Guided Cross-Modal Alignment for Robust 3D Detection](https://mlsanity.com/paper/2507.16861/) (2025)
- [Progressive Bird's Eye View Perception for Safety-Critical Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2508.07560/) (2025)
- [MambaFusion: Adaptive State-Space Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2602.08126/) (2026)
- [SGV3D:Towards Scenario Generalization for Vision-based Roadside 3D Object Detection](https://mlsanity.com/paper/2401.16110/) (2024)
- [From Dataset to Real-world: General 3D Object Detection via Generalized Cross-domain Few-shot Learning](https://mlsanity.com/paper/2503.06282/) (2025)
- [Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection](https://mlsanity.com/paper/2603.07486/) (2026)
- [GraphBEV++: Multi-Modal Feature Alignment for Autonomous Driving](https://mlsanity.com/paper/2606.16354/) (2026)
- [Towards robust multimodal 3D object detection via visual foundation models](https://mlsanity.com/paper/2609.23541/) (2026)

## Abstract (arXiv)

Multi-modal 3D object detectors are dedicated to exploring secure and reliable perception systems for autonomous driving (AD).Although achieving state-of-the-art (SOTA) performance on clean benchmark datasets, they tend to overlook the complexity and harsh conditions of real-world environments. With the emergence of visual foundation models (VFMs), opportunities and challenges are presented for improving the robustness and generalization of multi-modal 3D object detection in AD. Therefore, we propose RoboFusion, a robust framework that leverages VFMs like SAM to tackle out-of-distribution (OOD) noise scenarios. We first adapt the original SAM for AD scenarios named SAM-AD. To align SAM or SAM-AD with multi-modal methods, we then introduce AD-FPN for upsampling the image features extracted by SAM. We employ wavelet decomposition to denoise the depth-guided images for further noise reduction and weather interference. At last, we employ self-attention mechanisms to adaptively reweight the fused features, enhancing informative features while suppressing excess noise. In summary, RoboFusion significantly reduces noise by leveraging the generalization and robustness of VFMs, thereby enhancing the resilience of multi-modal 3D object detection. Consequently, RoboFusion achieves SOTA performance in noisy scenarios, as demonstrated by the KITTI-C and nuScenes-C benchmarks. Code is available at https://github.com/adept-thu/RoboFusion.