# Camera-Lidar Integration: Probabilistic sensor fusion for semantic mapping

arXiv: https://arxiv.org/abs/2007.05490 · HTML: https://mlsanity.com/paper/2007.05490/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 22 of 22)

- [UniScene: Unified Occupancy-centric Driving Scene Generation](https://mlsanity.com/paper/2412.05435/) (2024)
- [WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmarks for Autonomous Driving on Water Surfaces](https://mlsanity.com/paper/2307.06505/) (2023)
- [Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation](https://mlsanity.com/paper/2110.11062/) (2021)
- [Deep Learning Based 3D Segmentation: A Survey](https://mlsanity.com/paper/2103.05423/) (2021)
- [Towards Knowledge-driven Autonomous Driving](https://mlsanity.com/paper/2312.04316/) (2023)
- [PanoFlow: Learning 360deg Optical Flow for Surrounding Temporal Understanding](https://mlsanity.com/paper/2202.13388/) (2022)
- [Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning](https://mlsanity.com/paper/2103.00868/) (2021)
- [MASS: Multi-Attentional Semantic Segmentation of LiDAR Data for Dense Top-View Understanding](https://mlsanity.com/paper/2107.00346/) (2021)
- [Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency](https://mlsanity.com/paper/2506.07497/) (2025)
- [See Eye to Eye: A Lidar-Agnostic 3D Detection Framework for Unsupervised Multi-Target Domain Adaptation](https://mlsanity.com/paper/2111.09450/) (2021)
- [Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning](https://mlsanity.com/paper/2206.10711/) (2022)
- [MS3D++: Ensemble of Experts for Multi-Source Unsupervised Domain Adaptation in 3D Object Detection](https://mlsanity.com/paper/2308.05988/) (2023)
- [DTCLMapper: Dual Temporal Consistent Learning for Vectorized HD Map Construction](https://mlsanity.com/paper/2405.05518/) (2024)
- [Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method](https://mlsanity.com/paper/2510.22973/) (2025)
- [CalibRefine: Deep Learning-Based Online Automatic Targetless LiDAR-Camera Calibration with Iterative and Attention-Driven Post-Refinement](https://mlsanity.com/paper/2502.17648/) (2025)
- [A Wearable Data Collection System for Studying Micro-Level E-Scooter Behavior in Naturalistic Road Environment](https://mlsanity.com/paper/2212.11979/) (2022)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)
- [Panoptic-CUDAL: Rural Australia Point Cloud Dataset in Rainy Conditions](https://mlsanity.com/paper/2503.16378/) (2025)
- [M2S-RoAD: Multi-Modal Semantic Segmentation for Road Damage Using Camera and LiDAR Data](https://mlsanity.com/paper/2504.10123/) (2025)
- [Artifacts Mapping: Multi-Modal Semantic Mapping for Object Detection and 3D Localization](https://mlsanity.com/paper/2307.01121/) (2023)
- [Multi-Modal Camera-Based Detection of Vulnerable Road Users](https://mlsanity.com/paper/2509.06333/) (2025)
- [ArborSplat: Online Semantic Gaussian Splatting SLAM for Orchards](https://mlsanity.com/paper/2609.26315/) (2026)

## Abstract (arXiv)

An automated vehicle operating in an urban environment must be able to perceive and recognise object/obstacles in a three-dimensional world while navigating in a constantly changing environment. In order to plan and execute accurate sophisticated driving maneuvers, a high-level contextual understanding of the surroundings is essential. Due to the recent progress in image processing, it is now possible to obtain high definition semantic information in 2D from monocular cameras, though cameras cannot reliably provide the highly accurate 3D information provided by lasers. The fusion of these two sensor modalities can overcome the shortcomings of each individual sensor, though there are a number of important challenges that need to be addressed in a probabilistic manner. In this paper, we address the common, yet challenging, lidar/camera/semantic fusion problems which are seldom approached in a wholly probabilistic manner. Our approach is capable of using a multi-sensor platform to build a three-dimensional semantic voxelized map that considers the uncertainty of all of the processes involved. We present a probabilistic pipeline that incorporates uncertainties from the sensor readings (cameras, lidar, IMU and wheel encoders), compensation for the motion of the vehicle, and heuristic label probabilities for the semantic images. We also present a novel and efficient viewpoint validation algorithm to check for occlusions from the camera frames. A probabilistic projection is performed from the camera images to the lidar point cloud. Each labelled lidar scan then feeds into an octree map building algorithm that updates the class probabilities of the map voxels every time a new observation is available. We validate our approach using a set of qualitative and quantitative experimental tests on the USyd Dataset.