# A Survey of Deep Learning Based Radar and Vision Fusion for 3D Object Detection in Autonomous Driving arXiv: https://arxiv.org/abs/2406.00714 ยท HTML: https://mlsanity.com/paper/2406.00714/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. - Builds on 31 of its 50 references that are in the map. - First posted on arXiv 2 Jun 2024. ## Placed in - End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/) ## Builds on (most cited first, 30 of 31) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018) - [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017) - [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022) - [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021) - [CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2011.04841/) (2020) - [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022) - [K-Radar: 4D Radar Object Detection for Autonomous Driving in Various Weather Conditions](https://mlsanity.com/paper/2206.08171/) (2022) - [RADIATE: A Radar Dataset for Automotive Perception in Bad Weather](https://mlsanity.com/paper/2010.09076/) (2020) - [CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception](https://mlsanity.com/paper/2304.00670/) (2023) - [TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving](https://mlsanity.com/paper/2204.13483/) (2022) - [A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection](https://mlsanity.com/paper/2005.07431/) (2020) - [CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformer](https://mlsanity.com/paper/2209.06535/) (2022) - [RadarScenes: A Real-World Radar Point Cloud Data Set for Automotive Applications](https://mlsanity.com/paper/2104.02493/) (2021) - [Distant Vehicle Detection Using Radar and Vision](https://mlsanity.com/paper/1901.10951/) (2019) - [2D Car Detection in Radar Data with PointNets](https://mlsanity.com/paper/1904.08414/) (2019) - [Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perception](https://mlsanity.com/paper/2403.07746/) (2024) - [RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehicles](https://mlsanity.com/paper/1905.00526/) (2019) - [CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection](https://mlsanity.com/paper/2210.09267/) (2022) - [MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review](https://mlsanity.com/paper/2108.03004/) (2021) - [RCM-Fusion: Radar-Camera Multi-Level Fusion for 3D Object Detection](https://mlsanity.com/paper/2307.10249/) (2023) - [MVFAN: Multi-View Feature Assisted Network for 4D Radar Object Detection](https://mlsanity.com/paper/2310.16389/) (2023) - [RadarCam-Depth: Radar-Camera Fusion for Depth Estimation with Learned Metric Scale](https://mlsanity.com/paper/2401.04325/) (2024) - [HVDetFusion: A Simple and Robust Camera-Radar Fusion Framework](https://mlsanity.com/paper/2307.11323/) (2023) - [Vision-RADAR fusion for Robotics BEV Detections: A Survey](https://mlsanity.com/paper/2302.06643/) (2023) - [4DRVO-Net: Deep 4D Radar-Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion](https://mlsanity.com/paper/2308.06573/) (2023) - [CenterRadarNet: Joint 3D Object Detection and Tracking Framework using 4D FMCW Radar](https://mlsanity.com/paper/2311.01423/) (2023) ## Built on by (most cited first, 3 of 3) - [3DTMDet: A Dual-Path Synergy Network of Transformer and SSM for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2605.15546/) (2026) - [Understanding Cross-Sensor Feature Variations for Generalizable 3D Perception](https://mlsanity.com/paper/2606.11573/) (2026) - [Radar Guided Camera Verification for Automatic Emergency Braking Rethinking Object Detection in Radar Camera Fusion](https://mlsanity.com/paper/2606.27556/) (2026) ## Abstract (arXiv) With the rapid advancement of autonomous driving technology, there is a growing need for enhanced safety and efficiency in the automatic environmental perception of vehicles during their operation. In modern vehicle setups, cameras and mmWave radar (radar), being the most extensively employed sensors, demonstrate complementary characteristics, inherently rendering them conducive to fusion and facilitating the achievement of both robust performance and cost-effectiveness. This paper focuses on a comprehensive survey of radar-vision (RV) fusion based on deep learning methods for 3D object detection in autonomous driving. We offer a comprehensive overview of each RV fusion category, specifically those employing region of interest (ROI) fusion and end-to-end fusion strategies. As the most promising fusion strategy at present, we provide a deeper classification of end-to-end fusion methods, including those 3D bounding box prediction based and BEV based approaches. Moreover, aligning with recent advancements, we delineate the latest information on 4D radar and its cutting-edge applications in autonomous vehicles (AVs). Finally, we present the possible future trends of RV fusion and summarize this paper.