# Fail-Safe Human Detection for Drones Using a Multi-Modal Curriculum Learning Approach

arXiv: https://arxiv.org/abs/2109.13666 · HTML: https://mlsanity.com/paper/2109.13666/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [Semi-Supervised Deep Learning for Monocular Depth Map Prediction](https://mlsanity.com/paper/1702.02706/) (2017)
- [Learning Monocular Dense Depth from Events](https://mlsanity.com/paper/2010.08350/) (2020)
- [A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection](https://mlsanity.com/paper/2005.07431/) (2020)
- [CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations](https://mlsanity.com/paper/2005.01456/) (2020)
- [An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks](https://mlsanity.com/paper/2102.10423/) (2021)
- [RAMP-CNN: A Novel Neural Network for Enhanced Automotive Radar Object Recognition](https://mlsanity.com/paper/2011.08981/) (2020)
- [Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles](https://mlsanity.com/paper/2009.08428/) (2020)
- [Satellite Imagery Multiscale Rapid Detection with Windowed Networks](https://mlsanity.com/paper/1809.09978/) (2018)
- [A Low-Complexity Radar Detector Outperforming OS-CFAR for Indoor Drone Obstacle Avoidance](https://mlsanity.com/paper/2107.07250/) (2021)

## Built on by (most cited first, 7 of 7)

- [Fusing Event-based Camera and Radar for SLAM Using Spiking Neural Networks with Continual STDP Learning](https://mlsanity.com/paper/2210.04236/) (2022)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Resource-Efficient Gesture Recognition using Low-Resolution Thermal Camera via Spiking Neural Networks and Sparse Segmentation](https://mlsanity.com/paper/2401.06563/) (2024)
- [Continuously Learning to Detect People on the Fly: A Bio-inspired Visual System for Drones](https://mlsanity.com/paper/2202.08023/) (2022)
- [Rotational Odometry using Ultra Low Resolution Thermal Cameras](https://mlsanity.com/paper/2411.01227/) (2024)
- [Federated Learning of Low-Rank One-Shot Image Detection Models in Edge Devices with Scalable Accuracy and Compute Complexity](https://mlsanity.com/paper/2504.16515/) (2025)
- [Deep Fusion of Ultra-Low-Resolution Thermal Camera and Gyroscope Data for Lighting-Robust and Compute-Efficient Rotational Odometry](https://mlsanity.com/paper/2506.12536/) (2025)

## Abstract (arXiv)

Drones are currently being explored for safety-critical applications where human agents are expected to evolve in their vicinity. In such applications, robust people avoidance must be provided by fusing a number of sensing modalities in order to avoid collisions. Currently however, people detection systems used on drones are solely based on standard cameras besides an emerging number of works discussing the fusion of imaging and event-based cameras. On the other hand, radar-based systems provide up-most robustness towards environmental conditions but do not provide complete information on their own and have mainly been investigated in automotive contexts, not for drones. In order to enable the fusion of radars with both event-based and standard cameras, we present KUL-UAVSAFE, a first-of-its-kind dataset for the study of safety-critical people detection by drones. In addition, we propose a baseline CNN architecture with cross-fusion highways and introduce a curriculum learning strategy for multi-modal data termed SAUL, which greatly enhances the robustness of the system towards hard RGB failures and provides a significant gain of 15% in peak F1 score compared to the use of BlackIn, previously proposed for cross-fusion networks. We demonstrate the real-time performance and feasibility of the approach by implementing the system in an edge-computing unit. We release our dataset and additional material in the project home page.