# Real-Time Multi-Modal Semantic Fusion on Unmanned Aerial Vehicles with Label Propagation for Cross-Domain Adaptation

arXiv: https://arxiv.org/abs/2210.09739 · HTML: https://mlsanity.com/paper/2210.09739/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving](https://mlsanity.com/paper/2003.03653/) (2020)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation](https://mlsanity.com/paper/2004.01803/) (2020)
- [Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning](https://mlsanity.com/paper/1611.03631/) (2016)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/1911.12676/) (2019)
- [Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds](https://mlsanity.com/paper/2007.08488/) (2020)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)
- [One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation](https://mlsanity.com/paper/2104.02246/) (2021)
- [Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery](https://mlsanity.com/paper/1903.00268/) (2019)
- [MobileDets: Searching for Object Detection Architectures for Mobile Accelerators](https://mlsanity.com/paper/2004.14525/) (2020)
- [Sensor Fusion for Joint 3D Object Detection and Semantic Segmentation](https://mlsanity.com/paper/1904.11466/) (2019)
- [Deep Metric Transfer for Label Propagation with Limited Annotated Data](https://mlsanity.com/paper/1812.08781/) (2018)
- [weedNet: Dense Semantic Weed Classification Using Multispectral Images and MAV for Smart Farming](https://mlsanity.com/paper/1709.03329/) (2017)
- [Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mapping](https://mlsanity.com/paper/2111.11348/) (2021)
- [SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applications](https://mlsanity.com/paper/1907.11093/) (2019)

## Built on by (most cited first, 6 of 6)

- [LeAP: Consistent multi-domain 3D labeling using Foundation Models](https://mlsanity.com/paper/2502.03901/) (2025)
- [Self-Supervised Online Robot-Agnostic Traversability Estimation for Open-World Environments](https://mlsanity.com/paper/2605.28442/) (2026)
- [Artifacts Mapping: Multi-Modal Semantic Mapping for Object Detection and 3D Localization](https://mlsanity.com/paper/2307.01121/) (2023)
- [LiDAR-based Registration against Georeferenced Models for Globally Consistent Allocentric Maps](https://mlsanity.com/paper/2412.02533/) (2024)
- [See&Say: Vision Language Guided Safe Zone Detection for Autonomous Package Delivery Drones](https://mlsanity.com/paper/2604.13292/) (2026)
- [Joint Target-Less Intrinsic and Extrinsic Camera-LiDAR Calibration using Deep Point Correspondences](https://mlsanity.com/paper/2605.23397/) (2026)

## Abstract (arXiv)

Unmanned aerial vehicles (UAVs) equipped with multiple complementary sensors have tremendous potential for fast autonomous or remote-controlled semantic scene analysis, e.g., for disaster examination. Here, we propose a UAV system for real-time semantic inference and fusion of multiple sensor modalities. Semantic segmentation of LiDAR scans and RGB images, as well as object detection on RGB and thermal images, run online onboard the UAV computer using lightweight CNN architectures and embedded inference accelerators. We follow a late fusion approach where semantic information from multiple sensor modalities augments 3D point clouds and image segmentation masks while also generating an allocentric semantic map. Label propagation on the semantic map allows for sensor-specific adaptation with cross-modality and cross-domain supervision. Our system provides augmented semantic images and point clouds with ≈ 9 Hz. We evaluate the integrated system in real-world experiments in an urban environment and at a disaster test site.