# FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data

arXiv: https://arxiv.org/abs/1912.08487 · HTML: https://mlsanity.com/paper/1912.08487/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2019.

## Built on by (most cited first, 30 of 33)

- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)
- [MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2303.08600/) (2023)
- [OmniSat: Self-Supervised Modality Fusion for Earth Observation](https://mlsanity.com/paper/2404.08351/) (2024)
- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)
- [EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2106.15277/) (2021)
- [UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebase](https://mlsanity.com/paper/2309.05573/) (2023)
- [Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation](https://mlsanity.com/paper/2204.07548/) (2022)
- [Homogeneous Multi-modal Feature Fusion and Interaction for 3D Object Detection](https://mlsanity.com/paper/2210.09615/) (2022)
- [Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysis](https://mlsanity.com/paper/2210.04208/) (2022)
- [Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding](https://mlsanity.com/paper/2408.13024/) (2024)
- [Learning to Adapt SAM for Segmenting Cross-domain Point Clouds](https://mlsanity.com/paper/2310.08820/) (2023)
- [Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2410.22489/) (2024)
- [No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation](https://mlsanity.com/paper/2404.04050/) (2024)
- [WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments](https://mlsanity.com/paper/2312.15364/) (2023)
- [Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning](https://mlsanity.com/paper/2209.07774/) (2022)
- [Exploiting the Complementarity of 2D and 3D Networks to Address Domain-Shift in 3D Semantic Segmentation](https://mlsanity.com/paper/2304.02991/) (2023)
- [TASeg: Temporal Aggregation Network for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2407.09751/) (2024)
- [Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation](https://mlsanity.com/paper/2310.15676/) (2023)
- [How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation](https://mlsanity.com/paper/2505.18956/) (2025)
- [Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Network](https://mlsanity.com/paper/2308.04702/) (2023)
- [ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving](https://mlsanity.com/paper/2411.05311/) (2024)
- [Veila: Panoramic LiDAR Generation from a Monocular RGB Image](https://mlsanity.com/paper/2508.03690/) (2025)
- [Fusion-then-Distillation: Toward Cross-modal Positive Distillation for Domain Adaptive 3D Semantic Segmentation](https://mlsanity.com/paper/2410.19446/) (2024)
- [Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels](https://mlsanity.com/paper/2203.12655/) (2022)
- [DepthFormer: Multimodal Positional Encodings and Cross-Input Attention for Transformer-Based Segmentation Networks](https://mlsanity.com/paper/2211.04188/) (2022)
- [Normal Transformer: Extracting Surface Geometry from LiDAR Points Enhanced by Visual Semantics](https://mlsanity.com/paper/2211.10580/) (2022)
- [Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving](https://mlsanity.com/paper/2310.08826/) (2023)
- [Sparse Point Clouds Assisted Learned Image Compression](https://mlsanity.com/paper/2412.15752/) (2024)
- [Evaluation of Multimodal Semantic Segmentation using RGB-D Data](https://mlsanity.com/paper/2103.16758/) (2021)
- [FPPN: Future Pseudo-LiDAR Frame Prediction for Autonomous Driving](https://mlsanity.com/paper/2112.04401/) (2021)

## Abstract (arXiv)

We introduce a simple yet effective fusion method of LiDAR and RGB data to segment LiDAR point clouds. Utilizing the dense native range representation of a LiDAR sensor and the setup calibration, we establish point correspondences between the two input modalities. Subsequently, we are able to warp and fuse the features from one domain into the other. Therefore, we can jointly exploit information from both data sources within one single network. To show the merit of our method, we extend SqueezeSeg, a point cloud segmentation network, with an RGB feature branch and fuse it into the original structure. Our extension called FuseSeg leads to an improvement of up to 18% IoU on the KITTI benchmark. In addition to the improved accuracy, we also achieve real-time performance at 50 fps, five times as fast as the KITTI LiDAR data recording speed.