# LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1809.07941 · HTML: https://mlsanity.com/paper/1809.07941/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2018.

## Built on by (most cited first, 30 of 41)

- [Multi-Modal 3D Object Detection in Autonomous Driving: a Survey](https://mlsanity.com/paper/2106.12735/) (2021)
- [ONCE-3DLanes: Building Monocular 3D Lane Detection](https://mlsanity.com/paper/2205.00301/) (2022)
- [Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception](https://mlsanity.com/paper/2308.16714/) (2023)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving](https://mlsanity.com/paper/2103.03678/) (2021)
- [Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms](https://mlsanity.com/paper/2103.02433/) (2021)
- [SNE-RoadSeg+: Rethinking Depth-Normal Translation and Deep Supervision for Freespace Detection](https://mlsanity.com/paper/2107.14599/) (2021)
- [MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review](https://mlsanity.com/paper/2108.03004/) (2021)
- [Fast Road Segmentation via Uncertainty-aware Symmetric Network](https://mlsanity.com/paper/2203.04537/) (2022)
- [Analyzing Infrastructure LiDAR Placement with Realistic LiDAR Simulation Library](https://mlsanity.com/paper/2211.15975/) (2022)
- [WS-3D-Lane: Weakly Supervised 3D Lane Detection With 2D Lane Labels](https://mlsanity.com/paper/2209.11523/) (2022)
- [SNE-RoadSegV2: Advancing Heterogeneous Feature Fusion and Fallibility Awareness for Freespace Detection](https://mlsanity.com/paper/2402.18918/) (2024)
- [TrainSim: A Railway Simulation Framework for LiDAR and Camera Dataset Generation](https://mlsanity.com/paper/2302.14486/) (2023)
- [Optimizing the Placement of Roadside LiDARs for Autonomous Driving](https://mlsanity.com/paper/2310.07247/) (2023)
- [CLFT: Camera-LiDAR Fusion Transformer for Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2404.17793/) (2024)
- [GEM: Glare or Gloom, I Can Still See You -- End-to-End Multimodal Object Detection](https://mlsanity.com/paper/2102.12319/) (2021)
- [A novel multimodal fusion network based on a joint coding model for lane line segmentation](https://mlsanity.com/paper/2103.11114/) (2021)
- [Pseudo-LiDAR Based Road Detection](https://mlsanity.com/paper/2107.13279/) (2021)
- [Multi-Modal Multi-Task (3MT) Road Segmentation](https://mlsanity.com/paper/2308.11983/) (2023)
- [2D Feature Distillation for Weakly- and Semi-Supervised 3D Semantic Segmentation](https://mlsanity.com/paper/2311.15605/) (2023)
- [Panoptic Perception for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.15388/) (2024)
- [ISETAuto: Detecting vehicles with depth and radiance information](https://mlsanity.com/paper/2101.01843/) (2021)
- [SkipcrossNets: Adaptive Skip-cross Fusion for Road Detection](https://mlsanity.com/paper/2308.12863/) (2023)
- [Towards Safe Autonomy in Hybrid Traffic: Detecting Unpredictable Abnormal Behaviors of Human Drivers via Information Sharing](https://mlsanity.com/paper/2309.16716/) (2023)
- [Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)](https://mlsanity.com/paper/2408.00932/) (2024)
- [Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles](https://mlsanity.com/paper/2506.21885/) (2025)
- [SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving](https://mlsanity.com/paper/2510.18123/) (2025)
- [Composition and Application of Current Advanced Driving Assistance System: A Review](https://mlsanity.com/paper/2105.12348/) (2021)
- [Enabling Efficient Deep Convolutional Neural Network-based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2202.11231/) (2022)
- [Analyzing General-Purpose Deep-Learning Detection and Segmentation Models with Images from a Lidar as a Camera Sensor](https://mlsanity.com/paper/2203.04064/) (2022)

## Abstract (arXiv)

In this work, a deep learning approach has been developed to carry out road detection by fusing LIDAR point clouds and camera images. An unstructured and sparse point cloud is first projected onto the camera image plane and then upsampled to obtain a set of dense 2D images encoding spatial information. Several fully convolutional neural networks (FCNs) are then trained to carry out road detection, either by using data from a single sensor, or by using three fusion strategies: early, late, and the newly proposed cross fusion. Whereas in the former two fusion approaches, the integration of multimodal information is carried out at a predefined depth level, the cross fusion FCN is designed to directly learn from data where to integrate information; this is accomplished by using trainable cross connections between the LIDAR and the camera processing branches. To further highlight the benefits of using a multimodal system for road detection, a data set consisting of visually challenging scenes was extracted from driving sequences of the KITTI raw data set. It was then demonstrated that, as expected, a purely camera-based FCN severely underperforms on this data set. A multimodal system, on the other hand, is still able to provide high accuracy. Finally, the proposed cross fusion FCN was evaluated on the KITTI road benchmark where it achieved excellent performance, with a MaxF score of 96.03%, ranking it among the top-performing approaches.