# Cross-Modal Analysis of Human Detection for Robotics: An Industrial Case Study arXiv: https://arxiv.org/abs/2108.01495 ยท HTML: https://mlsanity.com/paper/2108.01495/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 43 references that are in the map. - First posted on arXiv 3 Aug 2021. ## Builds on (most cited first, 21 of 21) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018) - [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017) - [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016) - [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017) - [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019) - [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017) - [End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds](https://mlsanity.com/paper/1910.06528/) (2019) - [Deep Learning in Video Multi-Object Tracking: A Survey](https://mlsanity.com/paper/1907.12740/) (2019) - [3D Human Pose Estimation in RGBD Images for Robotic Task Learning](https://mlsanity.com/paper/1803.02622/) (2018) - [DR-SPAAM: A Spatial-Attention and Auto-regressive Model for Person Detection in 2D Range Data](https://mlsanity.com/paper/2004.14079/) (2020) - [Boosting LiDAR-based Semantic Labeling by Cross-Modal Training Data Generation](https://mlsanity.com/paper/1804.09915/) (2018) - [JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments](https://mlsanity.com/paper/1910.11792/) (2019) - [Deep Detection of People and their Mobility Aids for a Hospital Robot](https://mlsanity.com/paper/1708.00674/) (2017) - [LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images](https://mlsanity.com/paper/1910.13955/) (2019) - [Multisensor Online Transfer Learning for 3D LiDAR-based Human Detection with a Mobile Robot](https://mlsanity.com/paper/1801.04137/) (2018) - [Analyzing the Cross-Sensor Portability of Neural Network Architectures for LiDAR-based Semantic Labeling](https://mlsanity.com/paper/1907.02149/) (2019) - [Self-Supervised Person Detection in 2D Range Data using a Calibrated Camera](https://mlsanity.com/paper/2012.08890/) (2020) ## Built on by (most cited first, 5 of 5) - [SpectralWaste Dataset: Multimodal Data for Waste Sorting Automation](https://mlsanity.com/paper/2403.18033/) (2024) - [The Un-Kidnappable Robot: Acoustic Localization of Sneaking People](https://mlsanity.com/paper/2310.03743/) (2023) - [Acquisition of high-quality images for camera calibration in robotics applications via speech prompts](https://mlsanity.com/paper/2504.11031/) (2025) - [A Comparative Study of 3D Person Detection: Sensor Modalities and Robustness in Diverse Indoor and Outdoor Environments](https://mlsanity.com/paper/2602.05538/) (2026) - [Person Detection and Tracking from an Overhead Crane LiDAR](https://mlsanity.com/paper/2603.04938/) (2026) ## Abstract (arXiv) Advances in sensing and learning algorithms have led to increasingly mature solutions for human detection by robots, particularly in selected use-cases such as pedestrian detection for self-driving cars or close-range person detection in consumer settings. Despite this progress, the simple question "which sensor-algorithm combination is best suited for a person detection task at hand?" remains hard to answer. In this paper, we tackle this issue by conducting a systematic cross-modal analysis of sensor-algorithm combinations typically used in robotics. We compare the performance of state-of-the-art person detectors for 2D range data, 3D lidar, and RGB-D data as well as selected combinations thereof in a challenging industrial use-case. We further address the related problems of data scarcity in the industrial target domain, and that recent research on human detection in 3D point clouds has mostly focused on autonomous driving scenarios. To leverage these methodological advances for robotics applications, we utilize a simple, yet effective multi-sensor transfer learning strategy by extending a strong image-based RGB-D detector to provide cross-modal supervision for lidar detectors in the form of weak 3D bounding box labels. Our results show a large variance among the different approaches in terms of detection performance, generalization, frame rates and computational requirements. As our use-case contains difficulties representative for a wide range of service robot applications, we believe that these results point to relevant open challenges for further research and provide valuable support to practitioners for the design of their robot system.