# Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review

arXiv: https://arxiv.org/abs/2304.10410 · HTML: https://mlsanity.com/paper/2304.10410/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 190 of its 389 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Builds on (most cited first, 30 of 190)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)

## Built on by (most cited first, 30 of 58)

- [LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion](https://mlsanity.com/paper/2307.00724/) (2023)
- [WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces](https://mlsanity.com/paper/2307.06505/) (2023)
- [Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perception](https://mlsanity.com/paper/2403.07746/) (2024)
- [OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving](https://mlsanity.com/paper/2412.10734/) (2024)
- [Exploring Radar Data Representations in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2312.04861/) (2023)
- [Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception](https://mlsanity.com/paper/2501.15394/) (2025)
- [DPFT: Dual Perspective Fusion Transformer for Camera-Radar-based Object Detection](https://mlsanity.com/paper/2404.03015/) (2024)
- [Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension](https://mlsanity.com/paper/2405.12821/) (2024)
- [MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/2411.15016/) (2024)
- [WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar](https://mlsanity.com/paper/2403.12686/) (2024)
- [ASY-VRNet: Waterway Panoptic Driving Perception Model based on Asymmetric Fair Fusion of Vision and 4D mmWave Radar](https://mlsanity.com/paper/2308.10287/) (2023)
- [Radars for Autonomous Driving: A Review of Deep Learning Methods and Challenges](https://mlsanity.com/paper/2306.09304/) (2023)
- [CR3DT: Camera-RADAR Fusion for 3D Detection and Tracking](https://mlsanity.com/paper/2403.15313/) (2024)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [RICCARDO: Radar Hit Prediction and Convolution for Camera-Radar 3D Object Detection](https://mlsanity.com/paper/2504.09086/) (2025)
- [UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detection](https://mlsanity.com/paper/2409.14751/) (2024)
- [ZFusion: An Effective Fuser of Camera and 4D Radar for 3D Object Perception in Autonomous Driving](https://mlsanity.com/paper/2504.03438/) (2025)
- [Achelous++: Power-Oriented Water-Surface Panoptic Perception Framework on Edge Devices based on Vision-Radar Fusion and Pruning of Heterogeneous Modalities](https://mlsanity.com/paper/2312.08851/) (2023)
- [USVTrack: USV-Based 4D Radar-Camera Tracking Dataset for Autonomous Driving in Inland Waterways](https://mlsanity.com/paper/2506.18737/) (2025)
- [Wavelet-based Multi-View Fusion of 4D Radar Tensor and Camera for Robust 3D Object Detection](https://mlsanity.com/paper/2512.22972/) (2025)
- [Fuzzy-NMS: Improving 3D Object Detection with Fuzzy Classification in NMS](https://mlsanity.com/paper/2310.13951/) (2023)
- [RASMD: RGB And SWIR Multispectral Driving Dataset for Robust Perception in Adverse Conditions](https://mlsanity.com/paper/2504.07603/) (2025)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [CaRaFFusion: Improving 2D Semantic Segmentation with Camera-Radar Point Cloud Fusion and Zero-Shot Image Inpainting](https://mlsanity.com/paper/2505.03679/) (2025)
- [SFGFusion: Surface Fitting Guided 3D Object Detection with 4D Radar and Camera Fusion](https://mlsanity.com/paper/2510.19215/) (2025)
- [Dynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splitting](https://mlsanity.com/paper/2511.19021/) (2025)
- [Large Model Enabled Embodied Intelligence for 6G Integrated Perception, Communication, and Computation Network](https://mlsanity.com/paper/2512.15109/) (2025)
- [4DRC-OCC: Robust Semantic Occupancy Prediction Through Fusion of 4D Radar and Camera](https://mlsanity.com/paper/2603.07794/) (2026)
- [Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervision](https://mlsanity.com/paper/2310.05202/) (2023)
- [A Resource Efficient Fusion Network for Object Detection in Bird's-Eye View using Camera and Raw Radar Data](https://mlsanity.com/paper/2411.13311/) (2024)

## Abstract (arXiv)

Driven by deep learning techniques, perception technology in autonomous driving has developed rapidly in recent years, enabling vehicles to accurately detect and interpret surrounding environment for safe and efficient navigation. To achieve accurate and robust perception capabilities, autonomous vehicles are often equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these fused sensors, radars and cameras enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. This review aims to provide a comprehensive guideline for radar-camera fusion, particularly concentrating on perception tasks related to object detection and semantic segmentation.Based on the principles of the radar and camera sensors, we delve into the data processing process and representations, followed by an in-depth analysis and summary of radar-camera fusion datasets. In the review of methodologies in radar-camera fusion, we address interrogative questions, including "why to fuse", "what to fuse", "where to fuse", "when to fuse", and "how to fuse", subsequently discussing various challenges and potential research directions within this domain. To ease the retrieval and comparison of datasets and fusion methods, we also provide an interactive website: https://radar-camera-fusion.github.io.