mlsanityData to 7 Oct 2026

Real-Time Multi-Modal Semantic Fusion on Unmanned Aerial Vehicles with Label Propagation for Cross-Domain Adaptation

arXiv 2210.09739 ↗First posted 18 Oct 2022cs.CV cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

Unmanned aerial vehicles (UAVs) equipped with multiple complementary sensors have tremendous potential for fast autonomous or remote-controlled semantic scene analysis, e.g., for disaster examination. Here, we propose a UAV system for real-time semantic inference and fusion of multiple sensor modalities. Semantic segmentation of LiDAR scans and RGB images, as well as object detection on RGB and thermal images, run online onboard the UAV computer using lightweight CNN architectures and embedded inference accelerators. We follow a late fusion approach where semantic information from multiple sensor modalities augments 3D point clouds and image segmentation masks while also generating an allocentric semantic map. Label propagation on the semantic map allows for sensor-specific adaptation with cross-modality and cross-domain supervision. Our system provides augmented semantic images and point clouds with ≈ 9 Hz. We evaluate the integrated system in real-world experiments in an urban environment and at a disaster test site.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

39 of 58 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  9. 2019Searching for MobileNetV3cited by 1,984
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  11. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  12. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 388
27 more
  1. 2020SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Drivingcited by 226
  2. 2019Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mappingcited by 181
  3. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  4. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  5. 2016Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planningcited by 148
  6. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 144
  7. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 134
  8. 2019xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentationcited by 129
  9. 2020Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Cloudscited by 116
  10. 2021SuMa++: Efficient LiDAR-based Semantic SLAMcited by 100
  11. 2021One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentationcited by 89
  12. 2019Volumetric Instance-Aware Semantic Mapping and 3D Object Discoverycited by 79
  13. 2020MobileDets: Searching for Object Detection Architectures for Mobile Acceleratorscited by 55
  14. 2019Sensor Fusion for Joint 3D Object Detection and Semantic Segmentationcited by 48
  15. 2018Deep Metric Transfer for Label Propagation with Limited Annotated Datacited by 33
  16. 2017weedNet: Dense Semantic Weed Classification Using Multispectral Images and MAV for Smart Farmingcited by 29
  17. 2021Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mappingcited by 25
  18. 2019SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applicationscited by 21
  19. 2020Domain Adaptation in LiDAR Semantic Segmentation by Aligning Class Distributionscited by 17
  20. 2018Recurrent-OctoMap: Learning State-based Map Refinement for Long-Term Semantic Mapping with 3D-Lidar Datacited by 16
  21. 2018Boosting LiDAR-based Semantic Labeling by Cross-Modal Training Data Generationcited by 14
  22. 2019MAVNet: an Effective Semantic Segmentation Micro-Network for MAV-based Taskscited by 10
  23. 2019Semi-Supervised Semantic Mapping through Label Propagation with Semantic Texture Meshescited by 9
  24. 2021Real-time Multi-Adaptive-Resolution-Surfel 6D LiDAR Odometry using Continuous-time Trajectory Optimizationcited by 9
  25. 2020Online Object-Oriented Semantic Mapping and Map Updatingcited by 6
  26. 2020Comparing View-Based and Map-Based Semantic Labelling in Real-Time SLAMcited by 4
  27. 2020Semantic sensor fusion: from camera to sparse lidar informationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 3.

6 papers.

  1. 2025LeAP: Consistent multi-domain 3D labeling using Foundation Modelscited by 8
  2. 2026Self-Supervised Online Robot-Agnostic Traversability Estimation for Open-World Environmentscited by 1
  3. 2023Artifacts Mapping: Multi-Modal Semantic Mapping for Object Detection and 3D Localizationcited by 0
  4. 2024LiDAR-based Registration against Georeferenced Models for Globally Consistent Allocentric Mapscited by 0
  5. 2026See&Say: Vision Language Guided Safe Zone Detection for Autonomous Package Delivery Dronescited by 0
  6. 2026Joint Target-Less Intrinsic and Extrinsic Camera-LiDAR Calibration using Deep Point Correspondencescited by 0