# Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

arXiv: https://arxiv.org/abs/2004.05224 · HTML: https://mlsanity.com/paper/2004.05224/

## Key facts

- Cited by 108 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 30 of 108)

- [Multi-modal Sensor Fusion for Auto Driving Perception: A Survey](https://mlsanity.com/paper/2202.02703/) (2022)
- [Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys](https://mlsanity.com/paper/2303.17220/) (2023)
- [Multi-Modal 3D Object Detection in Autonomous Driving: a Survey](https://mlsanity.com/paper/2106.12735/) (2021)
- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)
- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusion](https://mlsanity.com/paper/2111.14382/) (2021)
- [MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment](https://mlsanity.com/paper/2209.00244/) (2022)
- [Look Inside for More: Internal Spatial Modality Perception for 3D Anomaly Detection](https://mlsanity.com/paper/2412.13461/) (2024)
- [Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving](https://mlsanity.com/paper/2311.11056/) (2023)
- [Towards Robust Physical-world Backdoor Attacks on Lane Detection](https://mlsanity.com/paper/2405.05553/) (2024)
- [Milestones in Autonomous Driving and Intelligent Vehicles Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviors](https://mlsanity.com/paper/2305.11239/) (2023)
- [Near-field Perception for Low-Speed Vehicle Automation using Surround-view Fisheye Cameras](https://mlsanity.com/paper/2103.17001/) (2021)
- [End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent](https://mlsanity.com/paper/2204.05513/) (2022)
- [Self-supervised Learning for Pre-Training 3D Point Clouds: A Survey](https://mlsanity.com/paper/2305.04691/) (2023)
- [Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning](https://mlsanity.com/paper/2202.04241/) (2022)
- [3D-OAE: Occlusion Auto-Encoders for Self-Supervised Learning on Point Clouds](https://mlsanity.com/paper/2203.14084/) (2022)
- [Adversarial Attacks on Camera-LiDAR Models for 3D Car Detection](https://mlsanity.com/paper/2103.09448/) (2021)
- [POS-BERT: Point Cloud One-Stage BERT Pre-Training](https://mlsanity.com/paper/2204.00989/) (2022)
- [Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention](https://mlsanity.com/paper/2405.18295/) (2024)
- [Proactive Anomaly Detection for Robot Navigation with Multi-Sensor Fusion](https://mlsanity.com/paper/2204.01146/) (2022)
- [CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding](https://mlsanity.com/paper/2310.06214/) (2023)
- [X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-modal Knowledge Transfer](https://mlsanity.com/paper/2312.07378/) (2023)
- [SAM4UDASS: When SAM Meets Unsupervised Domain Adaptive Semantic Segmentation in Intelligent Vehicles](https://mlsanity.com/paper/2401.08604/) (2023)
- [Online,Target-Free LiDAR-Camera Extrinsic Calibration via Cross-Modal Mask Matching](https://mlsanity.com/paper/2404.18083/) (2024)
- [Efficient 3D Recognition with Event-driven Spike Sparse Convolution](https://mlsanity.com/paper/2412.07360/) (2024)
- [Neural Intrinsic Embedding for Non-rigid Point Cloud Matching](https://mlsanity.com/paper/2303.01038/) (2023)
- [Explicitly Guided Information Interaction Network for Cross-modal Point Cloud Completion](https://mlsanity.com/paper/2407.02887/) (2024)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes](https://mlsanity.com/paper/2109.00452/) (2021)
- [Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion](https://mlsanity.com/paper/2212.00564/) (2022)

## Abstract (arXiv)

Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this paper devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.