mlsanityData to 7 Oct 2026

Real-Time Multi-Modal Semantic Fusion on Unmanned Aerial Vehicles

arXiv 2108.06608 ↗First posted 14 Aug 2021cs.CV cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Unmanned aerial vehicles (UAVs) equipped with multiple complementary sensors have tremendous potential for fast autonomous or remote-controlled semantic scene analysis, e.g., for disaster examination. In this work, we propose a UAV system for real-time semantic inference and fusion of multiple sensor modalities. Semantic segmentation of LiDAR scans and RGB images, as well as object detection on RGB and thermal images, run online onboard the UAV computer using lightweight CNN architectures and embedded inference accelerators. We follow a late fusion approach where semantic information from multiple modalities augments 3D point clouds and image segmentation masks while also generating an allocentric semantic map. Our system provides augmented semantic images and point clouds with ≈,9,Hz. We evaluate the integrated system in real-world experiments in an urban environment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 58 references that are in the map. First posted on arXiv 14 Aug 2021.

Builds on

38 of 58 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  9. 2019Searching for MobileNetV3cited by 1,984
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  11. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  12. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 388
26 more
  1. 2020SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Drivingcited by 226
  2. 2019Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mappingcited by 181
  3. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  4. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  5. 2016Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planningcited by 148
  6. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 144
  7. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 134
  8. 2019xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentationcited by 129
  9. 2020Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Cloudscited by 116
  10. 2021SuMa++: Efficient LiDAR-based Semantic SLAMcited by 100
  11. 2021One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentationcited by 89
  12. 2019Volumetric Instance-Aware Semantic Mapping and 3D Object Discoverycited by 79
  13. 2020MobileDets: Searching for Object Detection Architectures for Mobile Acceleratorscited by 55
  14. 2019Sensor Fusion for Joint 3D Object Detection and Semantic Segmentationcited by 48
  15. 2018Deep Metric Transfer for Label Propagation with Limited Annotated Datacited by 33
  16. 2017weedNet: Dense Semantic Weed Classification Using Multispectral Images and MAV for Smart Farmingcited by 29
  17. 2019SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applicationscited by 21
  18. 2020Domain Adaptation in LiDAR Semantic Segmentation by Aligning Class Distributionscited by 17
  19. 2018Recurrent-OctoMap: Learning State-based Map Refinement for Long-Term Semantic Mapping with 3D-Lidar Datacited by 16
  20. 2018Boosting LiDAR-based Semantic Labeling by Cross-Modal Training Data Generationcited by 14
  21. 2019MAVNet: an Effective Semantic Segmentation Micro-Network for MAV-based Taskscited by 10
  22. 2019Semi-Supervised Semantic Mapping through Label Propagation with Semantic Texture Meshescited by 9
  23. 2021Real-time Multi-Adaptive-Resolution-Surfel 6D LiDAR Odometry using Continuous-time Trajectory Optimizationcited by 9
  24. 2020Online Object-Oriented Semantic Mapping and Map Updatingcited by 6
  25. 2020Comparing View-Based and Map-Based Semantic Labelling in Real-Time SLAMcited by 4
  26. 2020Semantic sensor fusion: from camera to sparse lidar informationcited by 2

Built on by

Not cited yet.