# RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving

arXiv: https://arxiv.org/abs/1906.00208 · HTML: https://mlsanity.com/paper/1906.00208/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2019.

## Built on by (most cited first, 30 of 39)

- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)
- [MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2204.12667/) (2022)
- [MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2303.08600/) (2023)
- [EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2106.15277/) (2021)
- [UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebase](https://mlsanity.com/paper/2309.05573/) (2023)
- [Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception](https://mlsanity.com/paper/2307.13929/) (2023)
- [Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving](https://mlsanity.com/paper/2105.12713/) (2021)
- [Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysis](https://mlsanity.com/paper/2210.04208/) (2022)
- [Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2410.22489/) (2024)
- [BEVDetNet: Bird's Eye View LiDAR Point Cloud based Real-time 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2104.10780/) (2021)
- [LiMoSeg: Real-time Bird's Eye View based LiDAR Motion Segmentation](https://mlsanity.com/paper/2111.04875/) (2021)
- [Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning](https://mlsanity.com/paper/2209.07774/) (2022)
- [TASeg: Temporal Aggregation Network for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2407.09751/) (2024)
- [Revisiting Modality Imbalance In Multimodal Pedestrian Detection](https://mlsanity.com/paper/2302.12589/) (2023)
- [MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2503.06700/) (2025)
- [CLFT: Camera-LiDAR Fusion Transformer for Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2404.17793/) (2024)
- [Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation](https://mlsanity.com/paper/2411.12177/) (2024)
- [CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation](https://mlsanity.com/paper/2307.04091/) (2023)
- [MambaST: A Plug-and-Play Cross-Spectral Spatial-Temporal Fuser for Efficient Pedestrian Detection](https://mlsanity.com/paper/2408.01037/) (2024)
- [ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving](https://mlsanity.com/paper/2411.05311/) (2024)
- [CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems](https://mlsanity.com/paper/2502.17821/) (2025)
- [Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards](https://mlsanity.com/paper/2208.02483/) (2022)
- [Fusion-then-Distillation: Toward Cross-modal Positive Distillation for Domain Adaptive 3D Semantic Segmentation](https://mlsanity.com/paper/2410.19446/) (2024)
- [NLOS Dies Twice: Challenges and Solutions of V2X for Cooperative Perception](https://mlsanity.com/paper/2307.06615/) (2023)
- [Fast and Accurate: Video Enhancement using Sparse Depth](https://mlsanity.com/paper/2103.08764/) (2021)
- [Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving](https://mlsanity.com/paper/2310.08826/) (2023)
- [MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation](https://mlsanity.com/paper/2509.18198/) (2025)
- [Review of Deep Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2302.06370/) (2023)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)

## Abstract (arXiv)

LiDAR has become a standard sensor for autonomous driving applications as they provide highly precise 3D point clouds. LiDAR is also robust for low-light scenarios at night-time or due to shadows where the performance of cameras is degraded. LiDAR perception is gradually becoming mature for algorithms including object detection and SLAM. However, semantic segmentation algorithm remains to be relatively less explored. Motivated by the fact that semantic segmentation is a mature algorithm on image data, we explore sensor fusion based 3D segmentation. Our main contribution is to convert the RGB image to a polar-grid mapping representation used for LiDAR and design early and mid-level fusion architectures. Additionally, we design a hybrid fusion architecture that combines both fusion algorithms. We evaluate our algorithm on KITTI dataset which provides segmentation annotation for cars, pedestrians and cyclists. We evaluate two state-of-the-art architectures namely SqueezeSeg and PointSeg and improve the mIoU score by 10 % in both cases relative to the LiDAR only baseline.