# Event Camera and LiDAR based Human Tracking for Adverse Lighting Conditions in Subterranean Environments

arXiv: https://arxiv.org/abs/2304.08908 · HTML: https://mlsanity.com/paper/2304.08908/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 9 of 9)

- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Direct LiDAR Odometry: Fast Localization with Dense Point Clouds](https://mlsanity.com/paper/2110.00605/) (2021)
- [CERBERUS: Autonomous Legged and Aerial Robotic Exploration in the Tunnel and Urban Circuits of the DARPA Subterranean Challenge](https://mlsanity.com/paper/2201.07067/) (2022)
- [LOCUS: A Multi-Sensor Lidar-Centric Solution for High-Precision Odometry and 3D Mapping in Real-Time](https://mlsanity.com/paper/2012.14447/) (2020)
- [L2E: Lasers to Events for 6-DoF Extrinsic Calibration of Lidars and Event Cameras](https://mlsanity.com/paper/2207.01009/) (2022)
- [Event Guided Depth Sensing](https://mlsanity.com/paper/2110.10505/) (2021)
- [DARE-SLAM: Degeneracy-Aware and Resilient Loop Closing in Perceptually-Degraded Environments](https://mlsanity.com/paper/2102.05117/) (2021)
- [A Simple and Efficient Algorithm for Nonlinear Model Predictive Control](https://mlsanity.com/paper/1709.06487/) (2017)
- [UAVs Beneath the Surface: Cooperative Autonomy for Subterranean Search and Rescue in DARPA SubT](https://mlsanity.com/paper/2206.08185/) (2022)

## Built on by (most cited first, 3 of 3)

- [LiDAR-Event Stereo Fusion with Hallucinations](https://mlsanity.com/paper/2408.04633/) (2024)
- [BOX3D: Lightweight Camera-LiDAR Fusion for 3D Object Detection and Localization](https://mlsanity.com/paper/2408.14941/) (2024)
- [DELTA: Dense Depth from Events and LiDAR using Transformer's Attention](https://mlsanity.com/paper/2505.02593/) (2025)

## Abstract (arXiv)

In this article, we propose a novel LiDAR and event camera fusion modality for subterranean (SubT) environments for fast and precise object and human detection in a wide variety of adverse lighting conditions, such as low or no light, high-contrast zones and in the presence of blinding light sources. In the proposed approach, information from the event camera and LiDAR are fused to localize a human or an object-of-interest in a robot's local frame. The local detection is then transformed into the inertial frame and used to set references for a Nonlinear Model Predictive Controller (NMPC) for reactive tracking of humans or objects in SubT environments. The proposed novel fusion uses intensity filtering and K-means clustering on the LiDAR point cloud and frequency filtering and connectivity clustering on the events induced in an event camera by the returning LiDAR beams. The centroids of the clusters in the event camera and LiDAR streams are then paired to localize reflective markers present on safety vests and signs in SubT environments. The efficacy of the proposed scheme has been experimentally validated in a real SubT environment (a mine) with a Pioneer 3AT mobile robot. The experimental results show real-time performance for human detection and the NMPC-based controller allows for reactive tracking of a human or object of interest, even in complete darkness.